diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index aa055924..267e2f17 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -45,6 +45,11 @@ jobs: cache: pip cache-dependency-path: requirements-docs.txt + - name: Check generated workflow documentation + run: | + python -m unittest discover -s scripts -p 'test_generate_workflow_docs.py' -v + python scripts/generate-workflow-docs.py --check + - name: Install docs dependencies run: python -m pip install -r requirements-docs.txt diff --git a/AGENTS.md b/AGENTS.md index 036194b0..d85c879b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,11 +40,11 @@ for every source change. | Task | Implementation | Documentation and repository skill | | --- | --- | --- | -| CLI, target routing, command compatibility | `pkg/cmd`, `pkg/target`, `pkg/target/host`, `pkg/options`, `pkg/app` | [Targets](docs/advanced/targets.md), [CLI](docs/reference/cli.md), [pipeline](skills/k8s-launch-kit-pipeline/SKILL.md) | +| CLI, target routing, command compatibility | `pkg/cmd`, `pkg/target`, `pkg/target/host`, `pkg/options`, `pkg/app` | [Targets](engineering/advanced/targets.md), [CLI](docs/reference/cli.md), [pipeline](skills/k8s-launch-kit-pipeline/SKILL.md) | | Configuration, defaults, CLI overrides | `pkg/config`, `pkg/configinput`, `pkg/configflags`, `pkg/resolve` | [Configuration](docs/reference/configuration.md), [config](skills/k8s-launch-kit-config/SKILL.md) | | Discovery and topology presets | `pkg/nicconfigdaemon`, `pkg/networkoperatorplugin/discovery*`, `pkg/presets` | [Discovery](docs/user/discovery.md), [presets](docs/user/presets.md), [discover](skills/k8s-launch-kit-discover/SKILL.md) | | Profiles, grouping, templates, networking | `pkg/profiles`, `profiles`, `pkg/networkoperatorplugin/templates.go`, `pkg/networkoperatorplugin/scopes.go`, `pkg/networkoperatorplugin/spectrumx` | [Generation](docs/advanced/generation.md), [heterogeneous clusters](docs/user/heterogeneous-clusters.md), [Spectrum-X](docs/user/spectrum-x.md), [generate](skills/k8s-launch-kit-generate/SKILL.md), [networking context](skills/k8s-network-engineer/SKILL.md) | -| Artifact parsing and lifecycle handoff | `pkg/bundle`, `pkg/app`, `pkg/target/host`, `pkg/networkoperatorplugin` | [Artifact bundle](docs/architecture/artifact-bundle.md), [architecture](docs/architecture/overview.md) | +| Artifact parsing and lifecycle handoff | `pkg/bundle`, `pkg/app`, `pkg/target/host`, `pkg/networkoperatorplugin` | [Artifact bundle](engineering/architecture/artifact-bundle.md), [architecture](engineering/architecture/overview.md) | | Deployment, preflight, readiness | `pkg/networkoperatorplugin/deploy*`, `pkg/networkoperatorplugin/preflight`, `pkg/networkoperatorplugin/crstate`, `pkg/networkoperatorplugin/helmclient`, `pkg/target/host/deploy.go` | [Deployment](docs/advanced/deployment.md), [deploy](skills/k8s-launch-kit-deploy/SKILL.md), [dry run](skills/k8s-launch-kit-dryrun/SKILL.md) | | Validation, connectivity, reports | `pkg/target/host/validate*`, `pkg/networkoperatorplugin/connectivity`, `pkg/networkoperatorplugin/crstate` | [Validation](docs/user/validation.md), [validate](skills/k8s-launch-kit-validate/SKILL.md) | | OpenShift flavor | `pkg/config/flavor.go`, `pkg/networkoperatorplugin/ocp_config.go`, flavor-specific profiles and lifecycle code | [OpenShift](docs/user/openshift.md); relevant lifecycle skill's OpenShift section | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b27b8cea..5d5c574c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -57,12 +57,24 @@ the [documentation impact map](AGENTS.md#required-documentation-updates) to identify affected guides, README examples, architecture diagrams and skill playbooks, including their bundled references. +Keep `docs/` focused on operator procedures and reference material: what the +command does, where operations run, required inputs, outputs and recovery actions. +Keep architecture, implementation plans and contributor tooling in +[engineering documentation](engineering/README.md). Do not publish specific lab +clusters or development test-run history in user guides. + Before opening a PR: - Update existing sections and examples to describe the resulting behavior. - Check examples, links and any affected generated documentation. - Run `make build` and `python3 scripts/check-docs.py --binary build/l8k` to check documented flags, navigation and offline generation examples. +- For procedural workflow changes, edit the repository-owned definitions under + `openspec/workflows/` and regenerate with `python3 scripts/generate-workflow-docs.py`. + Run `python3 -m unittest discover -s scripts -p 'test_generate_workflow_docs.py'` + and `python3 scripts/generate-workflow-docs.py --check`. See the + [workflow source contract](openspec/workflows/README.md); these are documentation + inputs, not executable OpenSpec workflows or a runtime dependency. - For site changes, install `requirements-docs.txt` in an isolated Python environment and run `mkdocs build --strict`. - List documentation updates and validation in the PR description. If no diff --git a/README.md b/README.md index 9863d123..d7823f49 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,12 @@ Kubernetes Launch Kit (`l8k`) helps platform teams deploy NVIDIA networking on K Standard SR-IOV, RDMA-shared and host-device profiles enable NIC tuning by default; opt out with `--deploy-nic-configuration-template=false`. See [NIC configuration](docs/reference/configuration.md#nic-configuration-and-naming) for fabric-specific settings and operator ownership. -The current lifecycle uses the `host` target by default. The reserved `dpf` target does not have available lifecycle phases; see [target extension](docs/advanced/targets.md) for its implementation status. +The current lifecycle uses the `host` target by default. The `dpf` target provides an [experimental inventory discovery preview](docs/user/dpf-discovery.md); its provisioning phases remain unavailable. +DPF inventory can use explicit host/ARM SSH or BMC Redfish sources without an existing +management cluster. Inventory success is separate from provisioning readiness. +DPF BMC discovery verifies TLS by default; lab endpoints can explicitly opt in to +`tls.insecureSkipVerify: true`, recorded in discovery reports. See the preview guide +for its scope and security implications. Spectrum-X RA2.4 and RA3.1 support Network Operator 26.10 and newer catalogued releases; RA2.3 is limited to 26.7. Both use a doSPCX bundle and derive per-group platform @@ -166,7 +171,7 @@ Available Commands: version Print the version number Target Selection Flags: - --target string Deployment target: host (default) or dpf (reserved; phases are unavailable until the DPF driver is added) (default "host") + --target string Deployment target: host (default) or dpf (experimental inventory discovery) (default "host") Host Target Common Flags: --config-dir string Directory containing optional l8k-config.yaml and presets/ overrides @@ -255,14 +260,6 @@ Use the [container installation example](docs/user/installation.md#container). I Use the [OpenShift deployment guide](docs/user/openshift.md) for externally installed operators, flavor-specific resources, validation permissions, and qualification limits. -## Development +## Contributing -Read [AGENTS.md](AGENTS.md) and [CONTRIBUTING.md](CONTRIBUTING.md) before editing. The [architecture guide](docs/architecture/overview.md) covers the component map and Go discovery-library entry point. [Documentation publishing](docs/reference/docs-publishing.md) gives the build and consistency checks. The [extension contracts](openspec/README.md) cover shared behavior and conformance scenarios for new configuration, probes, profiles, resource kinds, and checks. - -### Library API - -See the [Go discovery-library integration](docs/architecture/overview.md#go-discovery-library) for the Helm-free public entry point, release and preset options, and logging/concurrency considerations. - -## Repository Automation - -Contributor automation and review workflow live in [AGENTS.md](AGENTS.md) and [CONTRIBUTING.md](CONTRIBUTING.md). +See [CONTRIBUTING.md](CONTRIBUTING.md) and the [engineering documentation](engineering/README.md) for development guidance. diff --git a/THIRD_PARTY_NOTICES b/THIRD_PARTY_NOTICES index 2c39dff0..6bdb95c3 100644 --- a/THIRD_PARTY_NOTICES +++ b/THIRD_PARTY_NOTICES @@ -9197,3 +9197,38 @@ Apache license: See the License for the specific language governing permissions and limitations under the License. + +--- +## github.com/gofrs/flock v0.13.0 + +Copyright (c) 2018-2025, The Gofrs +Copyright (c) 2015-2020, Tim Heckman +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of gofrs nor the names of its contributors may be used + to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +--- diff --git a/docs/advanced/concepts.md b/docs/advanced/concepts.md index a05792cc..b609ea75 100644 --- a/docs/advanced/concepts.md +++ b/docs/advanced/concepts.md @@ -11,8 +11,9 @@ The [planning guide](../user/profiles.md) uses these terms to choose a deployment. An `l8k` binary version, a Network Operator release line, a Spectrum-X RA version, a Kubernetes/OpenShift flavor, a deployment profile, and a topology preset describe different choices. Pinning one does not pin the -others. The `host` target covers the current lifecycle; the reserved `dpf` -target has no available phases. +others. The `host` target provides the networking lifecycle described here. +The separate `dpf` target provides [experimental inventory discovery](../user/dpf-workflows.md); +its provisioning phases are unavailable. ## Hardware diff --git a/docs/advanced/deployment.md b/docs/advanced/deployment.md index 0164f08b..ebaac77e 100644 --- a/docs/advanced/deployment.md +++ b/docs/advanced/deployment.md @@ -39,8 +39,7 @@ Launch Kit loads and checks the complete flat artifact directory before Helm installation, preflight remediation, or resource apply. Invalid YAML, missing resource identity fields, and duplicate declared resources stop deployment with the file and document number in the error. Only `values.yaml` is treated -as Helm values; rename `values.yml` or case variants to `values.yaml`. The -expected preflight inventory and applied objects come from that same snapshot. +as Helm values; rename `values.yml` or case variants to `values.yaml`. Keep the bundle unchanged while deployment runs. ## IPPool address collision guard @@ -98,7 +97,7 @@ authorizing overwrite. ## Helm Install Or Upgrade -When the bundle contains `values.yaml` and the selected release supplies a Helm repository URL, Launch Kit uses the Helm Go SDK to manage the `network-operator` release: +When the bundle contains `values.yaml` and the selected release supplies a Helm repository URL, Launch Kit manages the `network-operator` Helm release: - No release: install the selected chart in `networkOperator.namespace`. - Matching chart and values: no-op. @@ -229,7 +228,7 @@ resource with missing configuration fails rather than falling back to this mode. ## Terminal State -The shared resource-state registry classifies generated objects as: +Deployment reports each generated resource in one of these states: | State | Meaning | | --- | --- | @@ -245,9 +244,9 @@ CNI applied states when both desired and live specs leave those components external. It still requires ready status for every configured component in either spec and retains other component failures. Missing coverage waits; unrecognized or malformed evidence and policy-level reasons retain the original -aggregate verdict. This is the same classifier used by -[validation](../user/validation.md#external-networking-prerequisites); it does -not change deployment's ownership guards or controller-observation gate. +readiness result. [Validation](../user/validation.md#external-networking-prerequisites) +also checks external networking prerequisites. Deployment still checks resource +ownership and waits for controllers to observe the requested configuration. For `NicConfigurationTemplate` and `NicFirmwareTemplate`, Launch Kit first waits for the operator to publish matched device names in `status.nicDevices` and for that name set to reflect the current `nodeSelector`, NIC type, PCI-address, serial-number, and part-number selectors. It then evaluates only those `NicDevice` objects and waits for the corresponding `spec.configuration` or `spec.firmware` field to reflect the current template payload. A successful device condition is accepted only after its `observedGeneration` catches up with the `NicDevice` generation. A configuration template checks `FirmwareUpdateInProgress` only when the matched device carries `spec.firmware`; without a deployed firmware template, a stale firmware condition from an older device generation does not block configuration reconciliation. Other discovered NICs do not block on configuration or firmware state. Changed templates are also observation-gated before this status is accepted, so status left by an earlier generation cannot produce a false success. diff --git a/docs/advanced/generation.md b/docs/advanced/generation.md index fec8dcf2..b626b72d 100644 --- a/docs/advanced/generation.md +++ b/docs/advanced/generation.md @@ -94,7 +94,7 @@ Generated `SriovNetworkNodePolicy` and NV-IPAM `IPPool` resources carry the source identifiers selected for that artifact. Keep the annotation and `.l8k/resolved-config.yaml` together. Full merged buckets record all their sources; strict subsets record only their selected sources. Source identifiers -must be unique, including for direct library rendering. The established unnamed +must be unique. The established unnamed single-source configuration keeps its empty identifier and is represented by the explicit annotation `[""]`. It resolves only when the supplied configuration has exactly that one unnamed source; missing config or multiple sources fails rather @@ -133,12 +133,10 @@ with a deployed baseline. ## Bundle Layout Launch Kit renders and checks the complete artifact set before replacing the -selected plugin output directory. +selected deployment output directory. -Rendered files are structurally checked after ownership annotations and -before the output directory is cleaned. If this check fails, the previous -output remains in place. Successful output keeps the rendered bytes, and a -combined generate/deploy run uses the same checked snapshot. Network Operator +If a generated file fails validation, the previous output remains in place. +A combined generate/deploy run applies the generated bundle. Network Operator files use this layout: ```text diff --git a/docs/index.md b/docs/index.md index 69e51ed0..9ee63710 100644 --- a/docs/index.md +++ b/docs/index.md @@ -11,9 +11,9 @@ NVIDIA Kubernetes Launch Kit (`l8k`) helps platform teams configure NVIDIA netwo Use this site when you are deploying SR-IOV, RDMA shared-device, host-device, InfiniBand, or Spectrum-X networking. Start with [Plan your deployment](user/profiles.md) to choose a platform and profile, check what is qualified, and identify the site inputs and permissions you need. -The lifecycle commands operate on the `host` target by default. The reserved -`dpf` target has no available lifecycle phases; see the -[target extension contract](advanced/targets.md) if you are extending Launch Kit. +The lifecycle commands operate on the `host` target by default. The `dpf` target +provides an [experimental inventory discovery workflow](user/dpf-workflows.md) +for a separate DPU estate. DPF provisioning phases remain unavailable. ## Find Your Path @@ -27,7 +27,6 @@ The lifecycle commands operate on the `host` target by default. The reserved | Platform engineer managing mixed hardware | [Heterogeneous Clusters](user/heterogeneous-clusters.md) | | Spectrum-X operator | [Spectrum-X](user/spectrum-x.md) | | CI/CD or GitOps integrator | [Automation](integrator/automation.md) | -| Integrator adding an infrastructure target | [Target-aware CLI](advanced/targets.md) | | AI agent integrator | [AI Skills](integrator/ai-skills.md) | | Operator confirming a deployment is ready for use | [Validation](user/validation.md) | | Application owner connecting a workload | [Use the network in an application](user/workloads.md) | diff --git a/docs/integrator/ai-skills.md b/docs/integrator/ai-skills.md index 6cdef776..6c1bee14 100644 --- a/docs/integrator/ai-skills.md +++ b/docs/integrator/ai-skills.md @@ -152,21 +152,3 @@ Acceptance and triage: Run the normal validation workflow. If it does not pass, retain the test DaemonSet, collect a sosreport, and identify the first failed stage. ``` - -## Developing Launch Kit - -For source changes, start with the repository's -[AGENTS.md](https://github.com/NVIDIA/k8s-launch-kit/blob/main/AGENTS.md). -It covers development workflow, package boundaries, verification and required -documentation updates. `CLAUDE.md` points to that shared guide. - -For extension work, consult the [contract index](https://github.com/NVIDIA/k8s-launch-kit/blob/main/openspec/README.md). It defines shared integration behavior and scenarios for configuration, discovery, lifecycle, platform, profile, artifacts, resource kinds, ownership, connectivity, and output. Read the applicable baseline exceptions; the specs do not imply that every scenario is already enforced. Update a contract when its shared promise changes, and update the affected user guide and skill alongside implementation. Routine conforming additions do not need a new spec or proposal. - -Use operational skills from the checkout being changed when testing CLI -workflows. Source development can use `make build` and `./build/l8k` from the -repository root; a global installation and cluster access are not prerequisites. - -## Maintaining Skills - -Update relevant documentation sections, the corresponding skill and its bundled -references in the same PR whenever a CLI workflow, flag, default, exit code, or safety requirement changes. Keep examples aligned with `l8k schema` and command help, and keep shared behavior in `k8s-launch-kit-shared` instead of duplicating it across every phase. diff --git a/docs/integrator/automation.md b/docs/integrator/automation.md index f8787895..fce462ca 100644 --- a/docs/integrator/automation.md +++ b/docs/integrator/automation.md @@ -33,7 +33,7 @@ Choose the owners before building a pipeline: who renders, who manages the Netwo 4. Create a separate apply set for the external controller. `values.yaml` is Helm input for the Helm owner; `.l8k/` is metadata for Launch Kit; `*example*.yaml` files are validation fixtures. Apply only approved operational Kubernetes resources in dependency order. If Helm is externally owned, set `networkOperator.skipHelmChart: true` before generation. Avoid a controller prune policy that could delete other cohorts or manually owned resources; compare the complete desired/live inventory. 5. After the external controller reports reconciliation, retrieve the **same complete artifact** and run `l8k validate --deployment-files ./deployment --wait 10m`. Omit `--user-config` so validation uses the saved effective configuration. Keep the HTML report and apply the [acceptance outcomes](../user/validation.md#acceptance-outcomes); process success alone can leave incomplete connectivity coverage. -The reproduction record should contain the CLI version/build, selected operator release, source configuration, every referenced profile/topology/workload file, preset or template revision/digest, all render options, complete generated bundle and sidecar, apply owner/revision, and validation report. Mutable local paths alone cannot reproduce a render. The site follows development `main`; use the documentation at the installed release's tag or commit when operating an older binary. +The reproduction record should contain the CLI version/build, selected operator release, source configuration, every referenced profile/topology/workload file, preset or template revision/digest, all render options, complete generated bundle and sidecar, apply owner/revision, and validation report. Mutable local paths alone cannot reproduce a render. Use documentation matching the installed CLI release. ## JSON Mode @@ -52,8 +52,7 @@ Output contracts differ by command: | `preset list`, `preset update`, `sosreport` | Text output; these commands do not provide a JSON success contract despite accepting the inherited flag. | Lifecycle JSON mode sends human-readable progress to stderr and auto-confirms -prompts. Structured error objects may be emitted on failures; early Cobra flag -parse errors can instead write text to stderr. Validation can emit partial +prompts. Structured error objects may be emitted on failures; invalid command-line arguments can instead write text to stderr. Validation can emit partial results before a failure. Always check the process exit status. `--yes` is root-only. Use `--output json` for non-interactive lifecycle diff --git a/docs/llms.txt b/docs/llms.txt index f42cc634..2880e386 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -18,6 +18,8 @@ This documentation covers the `l8k` CLI for deploying NVIDIA cloud-native networ ## Deploy - Discover cluster hardware: /k8s-launch-kit/user/discovery/ +- DPF discovery preview: /k8s-launch-kit/user/dpf-discovery/ +- DPF discovery workflows: /k8s-launch-kit/user/dpf-workflows/ - Generate and review manifests: /k8s-launch-kit/advanced/generation/ - Deploy manifests: /k8s-launch-kit/advanced/deployment/ - Deploy on OpenShift: /k8s-launch-kit/user/openshift/ @@ -42,15 +44,3 @@ This documentation covers the `l8k` CLI for deploying NVIDIA cloud-native networ - CLI reference: /k8s-launch-kit/reference/cli/ - Configuration reference: /k8s-launch-kit/reference/configuration/ - -## Contribute - -- Architecture: /k8s-launch-kit/architecture/overview/ -- Artifact bundle internals: /k8s-launch-kit/architecture/artifact-bundle/ -- Extend deployment targets: /k8s-launch-kit/advanced/targets/ -- Maintain and publish documentation: /k8s-launch-kit/reference/docs-publishing/ - -## Design plans and history - -- DPF integration roadmap (in progress): /k8s-launch-kit/architecture/dpf-integration-plan/ -- HostTarget migration plan (implemented history): /k8s-launch-kit/architecture/host-target-migration-plan/ diff --git a/docs/reference/cli.md b/docs/reference/cli.md index c55f45d5..aaa06930 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -6,8 +6,8 @@ SPDX-License-Identifier: Apache-2.0 # CLI Reference Run `l8k --help` for the authoritative flag list. Run `l8k schema` for machine-readable capabilities. -Config-backed flags include a `configPaths` list in schema output, derived -from the same field tags that register the flags and apply explicit values. +Config-backed flags include a `configPaths` list in schema output showing the +configuration fields they set. ## Commands @@ -32,10 +32,9 @@ with a source filename and document number where applicable. ## Target selection and flag ownership -Omitting `--target` selects `host`; adding `--target host` follows the same -code path. `dpf` is a recognized target name whose phases are unavailable in -this build. Selecting it returns validation exit code `2`. This explicit -capability error prevents DPF invocations from falling through to host logic. +Omitting `--target` selects `host`. `dpf` provides an +[experimental discovery preview](../user/dpf-discovery.md). Its other phases +are unavailable and return exit `2`; they do not run against the host cluster. Host-only flags are rejected when they are explicitly supplied for another target. Defaults are ignored by this check, including explicit-value flags @@ -54,7 +53,7 @@ flag groups and `l8k schema` for each flag's `targets` list. | `--flavor` | root, discover, generate, deploy, validate, clean | host | `k8s` or `ocp`, overriding config. Clean accepts the flag to reject OpenShift cleanup explicitly. | | `--output json` | all (inherited) | target-agnostic | Command-specific output; validation emits a stream, preset/sosreport success remains text, and standalone deploy has no success envelope. See [Automation](../integrator/automation.md#json-mode). | | `--yes`, `-y` | root pipeline only | target-agnostic | Auto-confirm root prompts. Subcommands reject this flag; lifecycle JSON mode auto-confirms. | -| `--quiet` | root pipeline | target-agnostic | Suppress informational output. | +| `--quiet` | root pipeline, discover | target-agnostic | Suppress informational output; discovery keeps the final result and errors. | | `--log-level` | all | target-agnostic | Enable `trace`, `debug`, `info`, `warn`, or `error` logging. `debug` shows structured progress; `trace` also shows bounded command output. | | `--log-file` | all | target-agnostic | Write logs to a file instead of `stderr`. | @@ -90,7 +89,8 @@ available throughout the command tree. | `--overwrite-existing` | — | — | yes | yes | — | — | | `--skip-network-operator-helm` | yes | — | yes | yes | yes | — | | `--deployment-files` | — | — | — | yes | yes | — | -| `--yes`, `--quiet` | yes | — | — | — | — | — | +| `--yes` | yes | — | — | — | — | — | +| `--quiet` | yes | yes | — | — | — | — | | Validation flags below | — | — | — | — | yes | — | | `--keep-helm-chart` | — | — | — | — | — | yes | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 4f8bea8f..69b67f0b 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -5,14 +5,13 @@ SPDX-License-Identifier: Apache-2.0 # Configuration File -`cluster-config.yaml` keeps its existing public YAML shape. Discovery writes -it, while generation reads it without modifying it. The repository-root file -is also the canonical lowest-precedence defaults source embedded in the -binary. Generation records the effective result in +`cluster-config.yaml` contains hardware inventory and deployment settings. +Discovery writes it; generation reads it without modifying it and fills omitted +values from the built-in defaults. Generation records the effective result in `/.l8k/resolved-config.yaml`; deploy and validate use that bundle metadata when no explicit `--user-config` is supplied. -The canonical default release is `26.7`. The `26.10` catalog entry selects +The default release is `26.7`. The `26.10` catalog entry selects Network Operator `v26.10.0-beta.1` when requested. A compatible Spectrum-X release selected during fresh discovery takes precedence over that default; the saved file contains the selected release and its matching catalog versions. An @@ -536,8 +535,8 @@ Deploy and validate monitor all matched `NicDevice` objects and require current `ConfigUpdateInProgress` conditions with `reason: UpdateSuccessful` and `status: "False"`. Missing, stale or partial status remains in progress; runtime or firmware configuration failures are reported. Confirm NCO/DOCA/RHCOS runtime -RoCE support for the selected release before enabling this option on OCP; offline -rendering and tests do not establish live qualification. +RoCE support for the selected release before enabling this option on OCP, +then validate connectivity on the selected workers. Spectrum-X `NicConfigurationTemplate.spec.nicSelector` is not a separate configuration input. Launch Kit renders one template per source hardware group @@ -649,7 +648,7 @@ Fresh discovery may merge compatible source groups during generation, but it kee | Field | Meaning | | --- | --- | -| `identifier` | Lowercase resource-name form of machine/GPU identity with complete `NVIDIA` segments removed and common machine segments shortened (`ThinkSystem` → `ts`, `PowerEdge` → `pe`), bounded to 30 bytes with balanced machine/GPU prefixes and a 6-character deterministic hash when needed, or `group-N` fallback. The Launch Kit machine node label uses the same value. | +| `identifier` | Saved group name used with `--groups` and the Launch Kit machine node label. Names are lowercase and may be shortened; copy the saved value rather than constructing it. Unresolved multiple groups use `group-N`. | | `machineType` / `gpuType` | Discovered hardware identity. | | `linkType` | Per-group `Ethernet` or `InfiniBand` result when fabric probes agree. | | `netplanManaged` | `true` when any worker has an NVIDIA PF selected by a netplan `match.macaddress` stanza with a non-empty `set-name`. Blocks generation when a `NicInterfaceNameTemplate` would configure the group. | diff --git a/docs/user/discovery.md b/docs/user/discovery.md index c15216d9..461e0556 100644 --- a/docs/user/discovery.md +++ b/docs/user/discovery.md @@ -48,93 +48,6 @@ interface prefixes in `clusterConfig[].spectrumX.interfaceNames`. These do not replace observed PF names. See [Spectrum-X naming](spectrum-x.md#vr-dual-nic-rails) for VR differences, custom settings and regeneration behavior. -## Read-only library discovery - -Library consumers that only need the inventory from an already-running NIC -Configuration Daemon can call `networkoperatorplugin.DiscoverReadOnly`. The -function reads -the existing daemon pods, nodes, and `NicDevice` resources and returns a -`LaunchKitConfig` without creating, patching, or deleting Kubernetes resources. -It locates the existing daemon by its DaemonSet owner in the configured -Network Operator namespace when `WithReadOnlyNetworkOperatorNamespace` is -provided. Without that option, it checks the standard Network Operator and -Launch Kit namespaces, preferring a usable Network Operator daemon when both -installations are present. -It returns `networkoperatorplugin.ErrNotInstalled` when the discovery daemon is -not already running. With the configured-namespace option, callers need pod -read access only in that namespace; otherwise they need pod-list access in all -fallback namespaces. Both paths need cluster-scoped read access to nodes and -all-namespaces read permission for the namespaced `NicDevice` resources. Use -`discovery.Discover` when the caller is responsible for bootstrapping the -temporary daemon. - -The function does not use node labels to guess whether a daemon node can -publish a `NicDevice`. If every Ready daemon node already has a device, no pod -execution is needed. If any device is missing, pass -`networkoperatorplugin.WithReadOnlyRESTConfig` with the REST config for the -cluster. The function then runs the existing read-only NIC and BlueField trust -probe in that node's daemon pod. It excludes nodes with no discoverable NIC or -only restricted BlueFields, and reports a missing device for a verified NIC -publisher. It returns inspection and permission errors instead of treating -them as absent hardware. This fallback needs `pods/exec` permission in the -daemon namespace. - -Without a REST config, discovery returns an error by default. A caller that -accepts an incomplete inventory can opt in with -`networkoperatorplugin.WithReadOnlyAssumeUnverifiedExcluded()`. Ready daemon -nodes without a `NicDevice` are then excluded and a warning reports that their -publisher status is unverified. This option does not suppress errors from a -probe attempted with a REST config. At least one `NicDevice` on a Ready daemon -node is still required to construct an inventory. - -For optional fabric enrichment, pass both -`networkoperatorplugin.WithReadOnlyRESTConfig(restConfig)` and -`networkoperatorplugin.WithReadOnlyFabricDiscovery()`. This explicitly permits -read-only execs in the existing daemon pods and needs `pods/exec` permission in -the selected daemon namespace. Supplying only a REST config retains the existing -publisher-verification behavior and does not enable fabric probes. The fabric -option without a REST config fails before cluster reads. - -Fabric enrichment checks every selected worker and east-west PF, including sibling -PFs represented by a collapsed rail. One batched command per worker confirms -PCI-local RDMA names, enumerates all ports and reads their configured `link_layer`. -Only complete, unanimous observations set a group's `linkType`; partial, denied, -unreadable or conflicting observations leave it unknown with diagnostics. Link -state and subnet-manager presence do not gate fabric detection. An InfiniBand PF -without a netdev gets a separate IPoIB host-provisioning warning. - -Normal discovery uses the same all-worker fabric check. Collection allows four -concurrent workers with a ten-second deadline per worker and no timeout retry. -Its optional worst-case budget is approximately `ceil(selected workers / 4) * 10s` -plus API/dispatch overhead, capped by the caller's deadline. Internal probe -failures leave fabric unknown; caller cancellation or deadline returns an error. -Neither optional enrichment nor partial-inventory opt-in suppresses publisher -inspection errors when REST credentials are supplied. These observations prove -coverage of a successfully enumerated sysfs snapshot, not absent kernel hardware -or deployment readiness. No host modules, node labels or Kubernetes resources are -changed by read-only discovery. - -GPU product/topology enrichment is separately opt-in: -`networkoperatorplugin.WithReadOnlyGPUTopology("gpu-operator")` requires the -actual GPU Operator namespace and `WithReadOnlyRESTConfig(restConfig)`. -Default calls and REST-only calls do not acquire GPU probes. This option reads -host PCI identities in existing NIC daemon pods and tries `nvidia-smi` there, -then uses a verified same-node GPU Operator `gpu-feature-discovery` container -when necessary. It does not run the netplan workflow or modify either daemon. -Callers need pod read/exec access in both namespaces and DaemonSet read access -in the GPU Operator namespace. Missing namespace or REST credentials fails -before cluster reads. Optional failures leave unavailable GPU fields unset; -caller cancellation remains an error. Supply and consume a `pkg/ui` context -sink to receive these warnings; the library's default sink is silent. - -By default, read-only discovery also fails if it cannot derive a node selector -that matches only a group's workers. Inventory-only callers can pass -`networkoperatorplugin.WithReadOnlyAllowMissingNodeSelector()` to return the -topology in that case. The affected group's `NodeSelector` is left empty and -`WorkerNodes` is authoritative; do not use that empty selector for workload -placement. This option does not relax errors for missing or unverified -`NicDevice` resources. - ## Review saved inventory Open `cluster-config.yaml` before generation. Confirm the intended workers and @@ -244,10 +157,9 @@ VR NetworkBay rails are finalized afterward. Collection uses up to four workers, with a 30-second budget per node and one bounded GFD lookup per invocation. Source failures do not retry. NIC PF/NUMA -and netplan reads remain separate in the NIC daemon. Local tests verify source, -coverage and failure handling; each actual driver/toolkit/image/runtime stack -still needs live qualification. An accepted topology observation does not -establish GPU or network deployment readiness. +and netplan reads run in the NIC daemon. Check the selected GPU driver, toolkit +and container image against your workload requirements. An accepted topology +observation does not establish GPU or network deployment readiness. ### VR NetworkBay rails @@ -390,16 +302,12 @@ Discovery also writes up to two Launch Kit-owned labels: | `nvidia.kubernetes-launch-kit.machine: ` | Selects one source hardware group using the same value persisted in `clusterConfig[].identifier`. | | `nvidia.kubernetes-launch-kit.gpu: ` | Selects compatible source groups that share a GPU type. | -The machine label and group `identifier` share one lowercase, vendor-free value -bounded to 30 bytes. The same normalization shortens common machine segments: -`ThinkSystem` becomes `ts` and `PowerEdge` becomes `pe`. Long identities keep -balanced machine/GPU prefixes plus a 6-character deterministic hash; unused -prefix space moves to the longer component. GPU-only labels preserve the -discovered case and `NVIDIA` segment, using the Kubernetes 63-byte limit and the -existing 8-character shortening hash. If the machine or GPU type cannot be -resolved, discovery preserves the unnamed identifier for a sole source, or uses -`group-N` for multiple sources, and writes only labels it can construct. Either -case still requires a proven exact selector. +The machine label and group `identifier` use the same lowercase value, shortened +when necessary. The GPU label preserves the discovered GPU name. Use the saved +`clusterConfig[].identifier` with `--groups` rather than constructing it yourself. +If the machine or GPU type cannot be resolved, a single group can have an empty +identifier; multiple groups use `group-N`. Every selected group still needs a +node selector that matches only its intended workers. Before writing labels, discovery checks all proposed source identities and worker memberships for collisions. It publishes selectors only after labeling succeeds @@ -444,7 +352,7 @@ This prevents out-of-band or DPU-facing NICs from consuming rail and NV-IPAM all By default, `--collapse-nic-rails=true` records one rail per physical NIC. Multi-plane PFs on one physical port collapse to the master PF, while a model explicitly identified as dual-port keeps one rail per port. -Use one rail per PF for development or legacy configurations: +Use one rail per PF when your deployment requires that layout: ```bash l8k discover \ @@ -488,9 +396,8 @@ There is no machine-only or any-GPU fallback. If no exact pair matches, discover Discovery waits up to five minutes for daemon pods. It can proceed with the Ready pods when other daemon pods are on nodes that are not Ready, or when the wait -timeout expires while the caller's context is still active. If the caller's -context is cancelled or reaches its deadline first, discovery returns that -context error instead of continuing with an expired context. Discovery stops if +timeout expires while the command is still active. Cancellation or a command +deadline stops discovery and reports the reason. Discovery stops if no daemon pod becomes Ready. Keep the namespace when investigating image pulls, scheduling, or daemon startup: @@ -549,3 +456,9 @@ install. Review [secondary-network ownership](../reference/configuration.md#seco before selecting explicit managed/external policy. Unsupported or host-only installers can remain undetected; Multus detection does not imply plugin-suite presence. + +## Independent DPF inventory + +For management-cluster and BMC discovery of the separate DPU estate, use the +[DPF discovery preview](dpf-discovery.md). Its configuration, artifacts and +qualification gates are independent of the Host workflow above. diff --git a/docs/user/dpf-discovery.md b/docs/user/dpf-discovery.md new file mode 100644 index 00000000..acf88d51 --- /dev/null +++ b/docs/user/dpf-discovery.md @@ -0,0 +1,455 @@ + + +# DPF discovery preview + +Start with the [step-by-step CLI workflows](dpf-workflows.md) for first inventory, +address-candidate lookup and refresh. This page provides the access and output reference. + +`l8k discover --target dpf` collects hardware inventory from explicitly selected +host/DPU Redfish or SSH endpoints. The DPF estate is independent of Host networking. +Inventory mode can run before Kubernetes exists; an explicitly supplied management +kubeconfig adds management observations and must work if supplied. + +This build exposes an **experimental inventory preview**. Inventory success means +that selected sources were read, physical identities and unique host/card mappings +were established, each DPU BMC account had verified administrator privileges, and cleanup/publication succeeded. It does **not** mean DPF +provisioning readiness or production hardware support. Missing optional card +metadata remains visible, including cards without serials. Hardware support and +deployment readiness checks do not block inventory collection. Overall deployment `acceptance` remains +`not-evaluated`, even when `discovery.outcome` is `complete` and the command exits zero. + +Explicitly select `discovery.probes.mode: inventory`. The default `management` +mode is unavailable in this preview; selecting it stops the command before +remote access. Inventory reports management topology when a kubeconfig is supplied, +but does not test management-network connectivity. DPF generate/deploy/validate +and the DPF pipeline are unavailable. + +## Inputs and invocation + +Supply explicit host endpoints and DPU BMC endpoints for a complete inventory. +Each endpoint selects exactly one source: Redfish (`bmcEndpoint`) or SSH (`ssh`). +A host may contain several selected or unselected cards. With no explicit DPU +endpoints, BlueField cards are derived from host observations; their BMC/ARM +endpoints remain unknown and discovery is partial with a nonzero exit status. ARM-only +DPU sources also leave the required BMC administrator check failed. With explicit DPU endpoints, every selected source must +succeed and its card identity must match exactly one host adapter. Sources are not +silently substituted after access failures. The CLI does not scan subnets. + +```yaml +management: + kubeconfigRef: + file: ./credentials/management.kubeconfig + context: management +credentials: + dpu-reader: + username: bmc-maintainer + passwordRef: + env: DPU_BMC_PASSWORD + host-reader: + username: bmc-reader + passwordRef: + env: HOST_BMC_PASSWORD +discovery: + dpus: + - bmcEndpoint: https://192.0.2.11 + credentialRef: dpu-reader + hosts: + - bmcEndpoint: https://192.0.2.21 + credentialRef: host-reader + probes: + mode: inventory +``` + +```bash +l8k discover --target dpf --user-config dpf-site.yaml \ + --save-cluster-config dpf-discovered.yaml --output json +``` + +The YAML is ordinary CLI configuration; collected facts live under `inventory`. +Unknown fields, duplicate +keys, wrong scalar types, ambiguous contexts and duplicate endpoints fail before +remote access. Lists and credentials stay in YAML; no password CLI flag exists. + +| Source | Values | +| --- | --- | +| Required site inputs | Nonempty host list and explicit DPU BMC endpoints with administrator credentials for completion; each selected endpoint and credential alias; username and password reference or SSH private-key reference | +| Defaultable | `dpf.release: 26.4.0`; context from a supplied kubeconfig; system certificate trust; SSH port `22`; SSH sudo `false`; collection timeout `10m`; request timeout `15s`; up to four endpoints queried at once (`maxConcurrentBMCs`) | +| Conditional | Kubeconfig for optional management observations; SSH known-hosts file; approved noninteractive sudo for protected reads; private Redfish CA; `systemURI` when a BMC exposes multiple systems | +| Observed | Physical-card/system identities, linked host adapters, named DPU firmware versions, administrator verification, runtime hardware/network facts and management nodes/version | +| Release defaults | Software bill of materials (BOM), artifact coordinates and documented minima for supported models; omitted minima remain unknown and do not establish deployment readiness | +| Optional lookup inputs | Explicit ARM/BMC MAC selectors; optional local `dnsmasq-leases` files. Lookup returns address hints, never identity or permission to connect | +| Later phases | Deployment profile, DPU API allocation, DHCP/service networks and controller authentication | + +The management kubeconfig must use verified HTTPS and certificate/key or bearer +credentials. File and embedded representations of the same credential/trust field are mutually +exclusive; choose token or tokenFile. Exec/auth-provider plugins, basic and anonymous access are rejected +in this version. Only the supplied kubeconfig is used. The BMC must support +standard Redfish session login with the supplied username/password and allow +the account to read inventory. Basic authentication and vendor-specific session +paths are unsupported. Discovery deletes its login session when finished; a cleanup +failure is reported. It does not guess passwords or repair credentials. + +DPU BMC administrator authorization is required during discovery. The CLI reads +its own session, the advertised standard AccountService account collection and +the matching local account's linked role. An advertised account URI whose final +component matches the session username is read first and its returned username +is verified; unrelated account records need not be readable. For numeric/opaque +account URIs, the bounded complete collection must identify one matching account. The account username must match the +session, and the account must expose `Enabled: true`, `Locked: false` and +`PasswordChangeRequired: false`. Its role must grant all five standard privileges: +`Login`, `ConfigureManager`, `ConfigureUsers`, `ConfigureComponents` and +`ConfigureSelf`. A custom role granting this set is accepted. A username of +`admin`, a role name alone or OEM-only privileges do not establish authorization. +Missing, denied, malformed or conflicting account/role evidence fails closed. + +The owned session's `Roles` must match the account's assigned role. If `Roles` +is absent, the CLI requires the session's `UserAuthenticationSource` to identify +the trusted local AccountService. A current `LocalAccountAuth: Enabled` setting +does not prove how an existing session authenticated and cannot replace this proof. +`LocalAccountAuth: Disabled` and explicitly external session sources are +unsupported. `Fallback` and `LocalFirst` require that explicit local session +source, because those modes can authenticate through external providers. +When mode/source information is absent, a matching effective session role is +required; a username collision or an absent LDAP setting is insufficient. +Malformed or inconsistent exposed fields remain unverified even when another +field could otherwise establish the binding. This proves observed Redfish +privileges and matching local-account state, not future write success or DPF +controller authentication. External-provider role mappings are not evaluated. + +Review `bmcAdministratorVerified` on each DPU, any `issues` entry for that field, and the required +`dpf.management.bmc-administrator` check. `BMC_ADMIN_REQUIRED` means the role lacks +administrator privileges; `BMC_ADMIN_ACCESS_DENIED` means required account/role +reads were denied; `BMC_ADMIN_UNVERIFIABLE` means evidence was missing, malformed, +untrusted or inconsistent; `BMC_ADMIN_ACCOUNT_RESTRICTED` means disabled or locked; +`BMC_PASSWORD_CHANGE_REQUIRED` means an administrator must handle the password +change separately. Discovery never rotates passwords, changes roles, or tests +privileges with a write. Physical identity evidence remains in partial output. +Host BMC accounts need inventory read access; the administrator gate applies to +DPU BMC accounts. BF4 B4240 identity requires the exact advertised NVIDIA service +product and board model plus trusted System/chassis links. Newer firmware exposes +matching system/card serials. Older firmware can expose the card serial on its +B4240 BMC component, reciprocally contained by the physical Card linked to the +same System. Absent System metadata stays absent; exposed conflicting identities +fail. These graph and identity properties are reread before success. Free-text +descriptions and similar model names cannot qualify other hardware. This identity +support does not establish release qualification. + +Passwords resolve only from `{env: NAME}` or `{file: path}`. Environment values +are verbatim. UTF-8 files lose at most one final LF/CRLF. File, SSH known-hosts and CA references +are relative to the input YAML and are rebased when saved elsewhere. Private files +are not copied. BMC endpoints require HTTPS; redirected connections are not supported. + +Environment references are convenient for an automation system that injects +secrets for one invocation. They are not required: environment values are +inherited by child processes and can be exposed by process inspection or +diagnostic dumps, depending on the operating system and permissions. Avoid +typing literal passwords in shell commands or saving them in shell startup files. + +For a local secret file or a file supplied by a secret manager, use: + +```yaml +credentials: + dpu-reader: + username: admin + passwordRef: + file: ./credentials/dpu-bmc-password +``` + +The file contains the password only, not YAML. Restrict the directory to its +owner (`0700`) and the file to its owner (`0600`), keep it outside version control, +and manage its lifecycle separately. A file reference avoids exporting the secret +to the process environment; it does not encrypt the file. Choose one `env` or +`file` reference per password. Discovery does not save resolved passwords in the +inventory or logs. + +For BMCs with untrusted certificates, explicitly opt in per endpoint: + +```yaml +discovery: + dpus: + - bmcEndpoint: https://192.0.2.11 + credentialRef: dpu-reader + tls: + insecureSkipVerify: true +``` + +This fragment extends the complete input above. `insecureSkipVerify` is a YAML +boolean, defaults to `false`, and skips certificate-chain and hostname verification +for CLI-to-BMC Redfish connections only. HTTPS remains required, but the +server is not authenticated and BMC credentials can be exposed to impersonation. +Prefer a verified certificate/CA through `caFile` when available. + +The same field is accepted under `discovery.tls` as a shared BMC policy. An +endpoint `tls` block replaces the entire shared policy; `{insecureSkipVerify: false}` +restores system-trust verification for that endpoint. Omitted/null values use the +policy's default. `caFile` and `insecureSkipVerify: true` cannot be combined in the +effective policy. TLS policies are independent even when endpoints share credentials. + +The CLI warns on stderr unless quiet output is requested. Your `discovery` input +retains the selected TLS policy; `inventory.scopeDigest` protects refresh against +a changed scope or trust policy. The inventory does not repeat the configured +endpoint/trust list as `requestedScope`. A selected policy does not establish +successful TLS or hardware verification. Use a fresh input without +saved `inventory` and a new output when changing the verification policy or custom +CA contents. This BMC option does not relax management kubeconfig TLS verification. + +## Optional SSH inventory in Zero Trust + +SSH collects observations without using host RShim or DPU Redfish. Run the CLI +from a management node that can reach the configured endpoints. This version uses +direct SSH; it does not read ambient SSH configuration, agents or ProxyJump rules. +Existing jump connectivity must be provided through the chosen execution location. + +```yaml +credentials: + host-os: + username: inventory + passwordRef: {env: HOST_SSH_PASSWORD} + arm-os: + username: ubuntu + privateKeyRef: {file: ./credentials/arm-key} +discovery: + hosts: + - credentialRef: host-os + ssh: + address: 192.0.2.21:22 + knownHostsFile: ./credentials/known_hosts + sudo: true + dpus: + - credentialRef: arm-os + ssh: + address: 192.0.2.11:22 + knownHostsFile: ./credentials/known_hosts + sudo: true + probes: {mode: inventory} +``` + +This example collects partial evidence without a management kubeconfig or Redfish +credential; it exits nonzero because DPU BMC administrator proof is missing. +For a complete inventory, select the DPU BMC through Redfish instead of ARM SSH. Each SSH +credential uses exactly one password reference or unencrypted private-key file +reference. SSH host keys must match the explicitly supplied known-hosts file; +unknown/changed keys fail. Establish that trust through your site's trusted +inventory/onboarding process. `tls.insecureSkipVerify` never applies to SSH. + +Host collection reports the host identity and NVIDIA network cards, with PCI, +driver and NUMA information where available. DPU identity must be consistent +across the available system and card records; conflicting serials prevent a +complete inventory. Interfaces, routes, DNS and local BMC LAN address are additional +observations. Required tools must already exist; see the [tool prerequisites](dpf-workflows.md#initial-inventory-prepare). +Discovery runs read commands, optionally with `sudo -n`. It installs nothing, +prompts for no sudo password and changes no privileges. + +ARM discovery needs a booted, reachable OS with sufficient permissions. Local +`ipmitool lan print` can report the BMC endpoint without Redfish. The endpoint is +saved as `bmcEndpoint` alongside `method: ssh`; discovery does not connect or send +credentials to that newly observed address. Missing optional tools/data are +reported in `issues` with the affected `field`, a stable `code`, safe `message` +and `tool` where applicable; required-source failures retain their cause too. It +never infers a BMC address from an ARM IP or a serial from a hostname. + +If an optional SSH command times out, its result remains unavailable. Before the +next read, Launch Kit reconnects to the same configured endpoint using the same +credentials and verified host keys; it does not retry the timed-out command. +Reconnection failures, failed final identity reads, changed identity, and the +overall discovery deadline still prevent a complete inventory. + +Host and ARM PCI addresses belong to different domains; correlation uses physical +serial identity, not matching PCI addresses. If a same-serial host adapter exposes +a PCI vendor other than NVIDIA (`15b3`), the required association fails with +`HOST_ADAPTER_VENDOR_CONFLICT`. Inspect the endpoint, serial and PCI evidence and +rerun with corrected sources; a matching serial cannot override contradictory +vendor identity. A missing vendor stays unknown and does not establish hardware +qualification. Other vendors remain in host inventory, and a contradictory +candidate is never discarded to make another same-serial candidate appear unique. +Host-reported identity alone does not +attest trusted DPU state. Sources and any conflicts stay in the output. +SSH observations do not satisfy the required DPU BMC administrator check. +Controller mTLS enrollment and provisioning readiness remain separate later-phase checks. + +## Software BOM and optional address lookup + +`dpf.release` selects the release's software BOM independently of live measurements. +Omitted/null selects the `26.4.0` default; explicit empty or unknown releases +are rejected. The resolved BOM is saved under `inventory.software`, with operator +and component coordinates, exact hardware models and BFB version/URL plus a +digest when known. Documented minima use named fields such as +`minimumNICFirmware` and `minimumHostMFT`. Omitted minima remain unknown in +software checks; they are not inherited from a newer catalog. Catalog source +references, revision lists and generic requirement objects are not saved. +The current catalog has exact BlueField-3 models `B3210`, `B3220` and `B3240`; generic family names and BlueField-4 do not +establish applicability. Desired BFB/component coordinates are not measured +versions. Only documented minimum versions are compatibility checks; unsupported +firmware requirements remain unknown. Model applicability and software checks do +not qualify hardware or enable DPF deployment profiles. + +On the supported BF4/B4240 Redfish reader, `inventory.dpus[].psid` +records a successfully verified PSID from the selected System's `AssetTag`, falling back to the +identity-selected chassis's `AssetTag`. Missing, null, empty and `N/A` values +remain unknown. Both values, when supplied, must be bounded ASCII identifiers +and agree exactly; the selected value keeps its original spelling. The +same identity-bound source fields are reread before the PSID becomes observed. +This is a Redfish fact obtained from authenticated `AssetTag` reads; +it does not create an SSH/runtime observation or infer a PSID from a model. + +Failures and missing PSIDs are recorded in `issues` with `field: psid`. +`PSID_NOT_EXPOSED` and `PSID_READER_UNSUPPORTED` mean this source supplies no PSID. +`PSID_INVALID`, `PSID_CONFLICT`, `PSID_CHANGED` and `PSID_RECHECK_FAILED` leave +no eligible `psid` value. These optional gaps preserve +physical inventory and discovery's existing acceptance policy. A downstream +consumer requiring PSID must require a single observed value associated with the +selected device and reject contradictory or invalid readings across its sources; +an absent or unsupported Redfish PSID may be supplied by an independently verified +SSH measurement, but a rejected or conflicting Redfish PSID must not be bypassed. +The PSID fact and eligibility contribute to the inventory fingerprint and refresh +diff. Release qualification and administrator authorization remain separate checks. + +SSH can report NIC firmware/PSID, MFT version, NVConfig and privilege observations +when the existing tools and permissions permit the read. Current and next-boot +configuration remain distinct. DPU Redfish firmware is grouped by named +component under `firmware`, such as `bmc`, `nic`, `cec` and `uefi`. Each version +retains activation when known; an absent activation does not mean running. Raw +vendor names, firmware resource dumps and source-reference bookkeeping are +omitted. A failed source, partial firmware collection, identity change or explicit version/activation conflict +cannot satisfy a minimum through that source. An independent valid SSH observation +can still satisfy a minimum unless explicit component conflicts remain. + +`discovery.addressLookup` optionally combines named ARM/BMC MAC selectors with local +`dnsmasq-leases` snapshots and eligible neighbors already read through selected SSH +sources. Results and source/device reasons are saved under `inventory.addressLookup`. +An address candidate is unverified and never receives credentials automatically. +Follow the [candidate procedure](dpf-workflows.md#address-candidates) to prepare +lookup inputs, interpret ambiguity and explicitly add a verified endpoint. + +## Observations, reports and refresh + +Normal output shows live status on stderr as work happens, followed by a final +summary on stdout. Each status line identifies the selected source and the +operation: session connection, administrator verification, inventory reads, +identity rechecks, cleanup, host/card matching or output publication. +`RUNNING` precedes the operation; `PASS` follows a successful check or completed +read; `FAIL` identifies a failed operation. `UNAVAILABLE` identifies an optional +fact that could not be measured. Optional-read summaries count unavailable facts; +the saved detailed report retains actionable fact failures and required check reasons. Completing a read does not mean +all optional facts were available. Concurrent endpoints can interleave whole +status lines. `--quiet` suppresses progress; it does not change acceptance. + +The final text report uses configured endpoint names, falling back to addresses, +and shows DPU model/serial, host association and BMC administrator status. It +separates problems that prevent discovery from prerequisites for provisioning +and checks not requested in this run. Identical later-stage findings are grouped +with their affected sources. An unqualified hardware combination means this +build has not established qualification; it is not a detected hardware fault. +An unsupported BOM model means software compatibility is unknown. Neither is +silently marked passed, and the JSON check states and exit rules are unchanged. + +For operation timing and source/check diagnostics, add `--log-level debug`. +Use `--log-level trace` for Redfish request categories/statuses and SSH +stage/tool metadata. Logging does not include credentials, session tokens, +request/response bodies or raw remote command output. Debug/trace logs go to +stderr unless `--log-file` selects a file; normal live progress remains on stderr. + +```bash +l8k discover --target dpf --user-config dpf-site.yaml \ + --save-cluster-config dpf-discovered.yaml --log-level debug +``` + +To watch progress while saving a machine-readable result, redirect only stdout: + +```bash +l8k discover --target dpf --user-config dpf-site.yaml \ + --save-cluster-config dpf-discovered.yaml --output json \ + --log-level debug > discover.json +``` + +Redirecting `2>discover.log` saves progress and logs to that file instead of +displaying them. `--output json` still emits one final JSON document on stdout. + +### Read the device results + +Both host and DPU records use named fields for available facts and `issues` for +failures, mismatches and unavailable facts. Each issue identifies its `field` and +`code`, with a safe `message` and `tool` when available. Successful reads do not +create issue entries or repeat `state: observed` beside every value. The report's +checks still record required successes and later-phase gaps. + +Review `id`, `serialNumber`, `model`, `psid` and `bmcAdministratorVerified` where +present. `id` is published only after the required physical identity rechecks +succeed; a retained serial or endpoint on a failed record is diagnostic context. +A missing fact stays unknown. `firmwareInventoryComplete: true` means DPU +firmware enumeration completed, not that every version is known or compatible. +`firmware` contains named component version lists; activation remains separate +from version. ERoT components stay separate from BMC firmware. + +For SSH sources, `runtime.mftVersion`, `runtime.boardProduct` and +`runtime.hardware.cards` hold measured values. Each card can have `psid`, +`firmwareVersion`, `nextBoot` settings and `hostPrivileges`. For example, +`nextBoot.numVFs: 0` is an explicit measured zero, not a missing value; next-boot +settings do not prove the running mode. Host RShim service state, when readable, +is under `runtime.hardware.rshim`. + +Host BMC discovery keeps system identity, PCI adapter identities and advertised +reset types for host-to-DPU mapping. It does not dump host firmware or manufacture +DPU-only component placeholders. Host/DPU output omits generic observation maps, +authentication/time placeholders, raw command strings and collection bookkeeping. + +New saved inventories use format `v1alpha3`. This CLI also reads legacy +`v1alpha1` and `v1alpha2` conservatively: absent state or verification does not +become successful identity or administrator proof. Keep previous outputs as +history and refresh through discovery rather than editing version tags or facts. + +### Internal identifiers in saved inventory + +The detailed YAML/JSON retains identifiers for automation and traceability: + +| Field | Meaning | +| --- | --- | +| Device `recordKey` | Deterministic key for a selected observation source; used to connect checks to that device record. Normal text shows its name/address instead. | +| `scopeDigest` | Digest of the selected scope and recorded trust policy, used to detect incompatible refresh inputs. | +| `fingerprint` | Digest of selected material facts, used to compare inventories. | + +These `sha256:` values are internal identifiers/comparison digests, not secrets, +signatures or independent proof that hardware is correct. + +The discovered YAML omits all `evidence`, `evidenceRef` and `evidenceRefs` +bookkeeping, including management, SSH, address-lookup and connectivity-probe +records. Named facts, measurements and field-specific issues remain available. +The separate JSON diagnostic report retains source details where applicable. +A refresh preserves source backups; removing bookkeeping alone does not report +a physical change. + +### Review and refresh the handoff + +Review each card's serial and its host association. Missing or conflicting identity +must be resolved before accepting the inventory; an IP address, PCI address or +CPU serial does not establish card identity. Cards with incomplete information +remain visible with their available PCI/slot details. Every requested endpoint +remains represented on failure, and missing firmware or authorization facts +remain unknown. Inventory completion does not establish hardware support +for DPF deployment. + +Complete discovery writes the requested YAML. Partial collection preserves an +existing handoff and writes `partial.yaml` plus `report.json` beneath +`.discovery//`. JSON stdout is one final DPF result, with +progress on stderr. Invalid command-line arguments are reported on stderr. +The report contains facts, checks, factual diffs, later-phase gaps, errors and +successfully written artifacts. Redfish errors retain sanitized method/resource, +HTTP status and observation/cleanup stage. Cleanup failures retain the exact +validated session URI when known, never its token. Verify ownership before manual +recovery; an unknown cleanup result is not success. Authentication at session +creation does not establish permission to read protected inventory. +A complete in-place refresh uses the same input/output path and saves +`source-before.yaml` before replacing the inventory. A partial refresh leaves the +original file unchanged. Avoid concurrent runs or edits to the same output, and +use real directories without symlink aliases. Follow the [refresh procedure](dpf-workflows.md#refresh-inventory) +when inputs, trust settings or an older saved inventory need to change. + +| Exit | Meaning | +| --- | --- | +| 0 | Explicit-mode checks, cleanup and publication completed | +| 2 | Invalid input, incomplete required identity/mapping or DPU BMC administrator evidence, unavailable management mode or proven mismatch | +| 3 | Required observation, authentication, TLS, API or remote cleanup failure | +| 1 | Cancellation or local output error | diff --git a/docs/user/dpf-workflows.md b/docs/user/dpf-workflows.md new file mode 100644 index 00000000..1370b4b2 --- /dev/null +++ b/docs/user/dpf-workflows.md @@ -0,0 +1,387 @@ + + +# DPF discovery workflows + +Use these procedures with the experimental `l8k discover --target dpf` CLI. The DPU estate is separate from Host Network Operator deployment. Start with **inventory mode**; management probe mode is unavailable in this preview. Discovery collects facts and writes a handoff. It does not create VMs, install Kubernetes or DPF, configure DPUServices, change firmware, enroll trust, or start a DPF deployment. + +For complete field and error details, see [DPF discovery reference](dpf-discovery.md). Each diagram follows the same steps as its procedure. Commands are examples to run deliberately from a machine with direct access to your selected endpoints. + +## Collect the first inventory {#initial-inventory} + +Use this workflow when you know the host access addresses. Explicit DPU BMC endpoints and verified administrator privileges are required for a complete inventory. Without explicit DPU endpoints, Launch Kit derives BlueField card records from host inventory and leaves their access bindings unknown; host-only or ARM-only discovery retains partial evidence and exits nonzero. + +```mermaid +flowchart TD + n1["CLI machine: choose sources and access inputs"] + n2["CLI and endpoint administrators: prepare access"] + n3["CLI machine: create the inventory configuration"] + n4["CLI machine: start discovery"] + n5["Selected host, DPU and BMC: read identity and facts"] + n6["CLI machine: match cards and assess results"] + n7["CLI machine: review inventory and reasons"] + n8["Endpoint owner or CLI operator: correct the cause"] + n9["CLI machine: retain the inventory and report"] + n1 -->|"Sources selected"| n2 + n2 -->|"Access prepared"| n3 + n3 -->|"Configuration ready"| n4 + n4 -->|"Collect from selected endpoints"| n5 + n5 -->|"Return facts and read failures to CLI"| n6 + n6 -->|"Review saved results"| n7 + n7 -->|"Complete inventory reviewed"| n9 + n7 -->|"Required failure or conflict"| n8 + n8 -->|"Retry after correction"| n2 +``` + +### 1. CLI machine: choose sources and access inputs {#initial-inventory-choose-inputs} + +Choose each source explicitly: a host operating system over SSH or its BMC over Redfish; choose the DPU BMC over Redfish for a complete run. DPU ARM SSH is available for partial evidence but cannot verify BMC administrator privileges. A selected source must succeed; Launch Kit does not silently replace failed Redfish access with SSH. Host SSH can supply host-to-card identity in Zero Trust when PCI/VPD access is available. ARM SSH needs a booted, reachable DPU operating system. Neither access path proves trusted DPU state. + +| Input | What you supply or allow Launch Kit to determine | +| --- | --- | +| Always required | At least one host endpoint and explicit DPU BMC endpoints with administrator credentials for completion; the source type; a credential alias, username, and password reference or SSH key reference for each selected endpoint. Addresses and credentials are not guessed. | +| Required for SSH | A verified `knownHostsFile`; direct connectivity from the CLI machine. SSH port defaults to `22`; ambient SSH config, agents and ProxyJump are not used. | +| Conditional | DPU ARM SSH for partial OS measurements; a private CA for Redfish when system trust is insufficient; `systemURI` if a BMC exposes several Systems. | +| Optional | A management kubeconfig for additional observations. Inventory needs none, but an explicitly supplied kubeconfig must work. | +| Defaultable | `dpf.release` defaults to `26.4.0`; Redfish certificate verification is enabled; SSH `sudo` defaults to `false`. Choose `discovery.probes.mode: inventory` explicitly because the default management mode is unavailable in this preview. | +| Discovered | Host/system and card identities, host-to-DPU associations, visible hardware/software/network observations, and optional local BMC address hints. | + +Use an account whose own session proves its effective administrator role. If session Roles is absent, local authentication must be established by an explicit trusted local session `UserAuthenticationSource`. A current `LocalAccountAuth: Enabled` policy alone cannot prove how an existing session authenticated. Disabled local authentication and explicitly external session sources are unsupported; Fallback/LocalFirst require the explicit local session source. Missing origin plus missing Roles cannot inherit privileges from a same-named local administrator. Discovery reports unverifiable proof as partial without replacing a complete handoff. + +Do not provide product versions as measured facts. The selected release supplies the software bill of materials (BOM); discovery measures installed versions separately. Omitted or null `dpf.release` uses the pinned default; an empty or unknown release is rejected. + +All network connections start from the machine where you run `l8k`. Host and DPU endpoints do not need to connect to each other for this workflow. + +| Connection from the CLI machine | Where the operation runs | What it provides | +| --- | --- | --- | +| SSH to the selected host | Host operating system | Host identity, installed card serials and available host/card facts | +| SSH to the selected DPU ARM address | DPU operating system | DPU identity, local hardware/network facts, and optionally its BMC address through local IPMI | +| HTTPS to a selected host or DPU BMC | That BMC's Redfish service | System/card identity, host PCI/reset information and available named DPU firmware versions | +| HTTPS to the management API, only if a kubeconfig is supplied | Existing management Kubernetes API | Cluster identity, version and nodes, plus optional storage and advertised DPF API information | +| No remote connection | CLI machine | Read input/trust/lease files, match identities, compare software requirements and save results | + +Choose SSH or Redfish separately for each endpoint. Selecting host Redfish does not run tools in the host OS. Selecting DPU SSH does not log in to DPU Redfish. The [collection step](#initial-inventory-collect) lists the operations for each choice. + +Next: [Sources selected](#initial-inventory-prepare). + +### 2. CLI and endpoint administrators: prepare access {#initial-inventory-prepare} + +Put passwords in an environment variable or a private file, and SSH private keys and known-hosts entries in files under your control. Reference them from YAML; do not put password values in command-line arguments. File references are relative to that YAML file. Verify SSH keys and Redfish CA trust through your normal provisioning process. + +For SSH inventory, ask the endpoint administrator to make the required read tools and permissions available before the run. Host identity/PCI collection uses tools such as `dmidecode` and `lspci`; DPU identity additionally uses `ipmitool`. Optional hardware checks use `mlxfwmanager`, `mlxconfig`, `mlxprivhost` and `systemctl`; network checks use existing `ip`, resolver and LLDP tools where available. Tool availability and access differ between host and ARM operating systems. + +Set `ssh.sudo: true` only where noninteractive elevated reads are authorized and configured. Launch Kit then uses `sudo -n`; it never prompts for a sudo password, installs tools, changes privileges or repairs device settings. Missing tools, denied permissions and unsupported or malformed outputs keep specific reason codes and explanations. Required identity failures stop a complete inventory; optional unknown observations remain visible for review. + +Prepare a DPU BMC account that is enabled, unlocked and does not require a password change, with all five standard Administrator privileges. A custom role with those privileges is accepted; names alone are insufficient. Discovery verifies the session, unique matching account and linked role using read requests. Handle password changes separately before rerunning; discovery never rotates passwords or tests privileges through writes. + +Passwords may use an environment reference supplied for the invocation or a `passwordRef.file` pointing to an owner-only UTF-8 password file. Environment variables are convenient for automation but can be inherited or exposed by diagnostics. File references avoid that export but remain plaintext files: keep them private and outside version control. Discovery saves references, never secret values. + +Next: [Access prepared](#initial-inventory-configure). + +### 3. CLI machine: create the inventory configuration {#initial-inventory-configure} + +Save this example as `dpf-site.yaml`, replacing the documentation address and credential paths with your host values. It uses host SSH and DPU BMC Redfish and requires no Kubernetes cluster. Ensure the listed files exist before running. + +```yaml +dpf: + release: 26.4.0 +credentials: + host-reader: + username: inventory + privateKeyRef: {file: ./credentials/host-key} + dpu-maintainer: + username: bmc-maintainer + passwordRef: {env: DPU_BMC_PASSWORD} +discovery: + hosts: + - credentialRef: host-reader + ssh: + address: 192.0.2.21:22 + knownHostsFile: ./credentials/known_hosts + sudo: true + dpus: + - bmcEndpoint: https://192.0.2.11 + credentialRef: dpu-maintainer + probes: + mode: inventory +``` + +For DPU ARM observations, use the SSH shape instead; ARM-only DPU input produces partial inventory with a nonzero exit because it cannot prove BMC administrator privileges. For Redfish, use `bmcEndpoint` and a password credential reference instead of `ssh`; see the [complete access examples](dpf-discovery.md#inputs-and-invocation). Each endpoint has exactly one access source. Plain YAML is the CLI input, not a Kubernetes object. Do not copy previously observed `inventory` facts into a new seed as user assertions. + +Next: [Configuration ready](#initial-inventory-run). + +### 4. CLI machine: start discovery {#initial-inventory-run} + +Choose a new output path for the first run. Run from a machine that can reach every selected endpoint and read the referenced files. + +```bash +l8k discover --target dpf --user-config dpf-site.yaml \ + --save-cluster-config dpf-discovered.yaml --output json \ + > discovery.json 2> discovery.log +``` + +Record the process exit status as well as the files. `discovery.json` contains the final result; progress and diagnostics go to `discovery.log`. The command creates its own run directory next to the requested YAML. It does not send credentials to addresses found during collection. Redfish uses a temporary login session and deletes that session; SSH runs read commands on explicitly selected endpoints. + +While this command runs, Launch Kit performs the collection and matching steps below automatically. You do not need to run the listed remote commands yourself. The example contacts the host over SSH and the explicitly selected DPU BMC over Redfish. Host-only input contacts no DPU endpoint and returns partial evidence with a nonzero exit. + +Watch live source/check status on stderr: RUNNING is emitted before the operation, followed by its result. Endpoint names/addresses distinguish concurrent work. To keep progress visible with JSON, redirect stdout only (`>discover.json`). Add `--log-level debug` for stage timing or `--log-level trace` for request/tool metadata; `2>discover.log` saves these messages instead of displaying them. Quiet suppresses progress. + +Next: [Collect from selected endpoints](#initial-inventory-collect). + +### 5. Selected host, DPU and BMC: read identity and facts {#initial-inventory-collect} + +Launch Kit uses the configured connection for each endpoint. Required identity reads and DPU BMC administrator verification must succeed to complete inventory. Optional reads add facts; a missing tool, denied operation or unsupported response leaves an explained gap instead of silently passing the check. After collecting optional facts, Launch Kit rereads identity to detect a changed or replaced device. If an optional SSH command times out, Launch Kit retains the reason and reconnects to the same configured endpoint with the same credentials and verified host keys before the next read. It does not retry the timed-out command. Failed reconnection, a failed identity reread or an expired overall deadline still prevents complete inventory. + +**On a host selected through SSH** + +| Read performed on the host | Purpose and result | Required or optional | +| --- | --- | --- | +| `dmidecode -s system-uuid` | Identifies the physical host. | Required | +| `lspci -Dnn -d 15b3:` and per-card `lspci -D -vv -s ` | Finds NVIDIA network cards, their PCI locations and available card serial/slot information. | PCI enumeration is required; a selected DPU needs a usable serial for host-to-card matching. Missing serials remain visible. | +| Linux PCI device files | Reads each function's network interface, driver and NUMA location. | Optional | +| `systemctl show rshim --property=LoadState --property=ActiveState` | Reports whether the host's RShim service is present and active. It does not start the service or use it to access a DPU. | Optional | + +**On a DPU ARM operating system selected through SSH** + +| Read performed on the DPU ARM OS | Purpose and result | Required or optional | +| --- | --- | --- | +| `dmidecode -s system-serial-number` and `ipmitool fru print 0` | Reads system and built-in card records; their serials must agree. Also records available model/board information. | Required for DPU identity | +| `ipmitool lan print` | Reads the local BMC's configured LAN address over local IPMI. | Optional | +| `lspci -Dnn -d 15b3:` and per-card `lspci -D -vv -s ` | Locates ARM-visible NVIDIA cards and available card serials before associating hardware readings with them. | Optional | +| `mlxprivhost -d q --full` | Reports host privilege restrictions, such as restricted access or disabled RShim, for a visible BlueField. | Optional | + +Local IPMI is an operation **from the DPU ARM OS to its local BMC**, not an HTTPS login from the CLI. The returned BMC address is recorded, but Launch Kit does not connect to it or send it credentials. Host and ARM PCI addresses are local to different operating systems and need not match. + +**On each selected host or DPU ARM OS, through SSH** + +| Read performed on that OS | Purpose and result | +| --- | --- | +| `mlxfwmanager --version` | Records the installed MFT tool version. | +| `mlxfwmanager --query -d ` | Queries the specific visible card's current NIC firmware and PSID. | +| `mlxconfig -d q ` | Reads `INTERNAL_CPU_OFFLOAD_ENGINE`, `INTERNAL_CPU_RSHIM` and `NUM_OF_VFS` on visible BlueFields. These results are next-boot settings, not proof of the running mode. | +| `hostname`, `cat /etc/os-release`, `ip -j address show`, `ip -j -4 route show` | Reports the OS, local addresses and IPv4 routes. | +| `ip -j link show`, `ip -j -d link show`, `ip -j neighbor show` | Reports MACs, bridge/member relationships, available link details and neighbors already known to that OS. It does not scan the subnet. | +| `cat /etc/resolv.conf`, `readlink -f /etc/resolv.conf`, `resolvectl dns`, `resolvectl domain` | Reports configured DNS servers, search domains, per-link DNS and the resolver configuration in use. | +| Selected `server=` and `no-resolv` settings in `/etc/dnsmasq.conf` and `/etc/dnsmasq.d/*.conf` | Reports local dnsmasq DNS forwarding configuration. This is separate from reading a DHCP lease snapshot on the CLI machine. | +| `lldpctl -f keyvalue` | Reports neighbors already available from the local LLDP service. | + +These hardware and network reads are optional. `` comes from that same OS's card inventory; Launch Kit queries each card separately. In Zero Trust, the host may be unable to query a restricted card even when `mlxfwmanager` is installed. That limitation is reported; it does not cause an automatic connection to the DPU. Tools must be installed on the OS where the command runs, not merely on the CLI machine. When configured, protected hardware reads use `sudo -n`; tools are never installed or settings changed by discovery. + +**On a selected host or DPU BMC, through HTTPS Redfish** + +The CLI creates a temporary session and reads system and card inventory. Host BMC reads locate cards associated with the selected host and retain advertised reset types; they do not collect host firmware/time/NTP or DPU-only component placeholders. DPU BMC reads identify that DPU, query available firmware components and verify the authenticated session account and linked role through AccountService. All five standard Administrator privileges and enabled/unlocked/no-password-change state are required; denied or unverifiable evidence fails closed. The supported BF4/B4240 reader also records an optional PSID from the selected System or identity-selected chassis AssetTag, with authenticated reads and a final reread. Missing PSID stays unknown; invalid, contradictory or changed fields remain unavailable for consumers requiring PSID. No PSID is guessed from the model. No SSH commands run in either OS for this choice. System/card identity is required; optional component versions may remain unknown. A firmware version is not assumed to be running unless the BMC explicitly identifies it as active. The CLI deletes its session afterward and reports any cleanup failure. It does not change passwords, roles, firmware, NTP, power or trust settings. + +**At the management Kubernetes API, if configured** + +The CLI reads the cluster identity from `kube-system`, the Kubernetes version and the node list, including control-plane membership and reported readiness. These reads are required when you supply a management kubeconfig. StorageClasses and advertised DPF API groups are additional optional reads. Inventory mode creates no Pods and performs no management-network connectivity tests. It does not install DPF or run commands inside Kubernetes nodes. + +Next: [Return facts and read failures to CLI](#initial-inventory-match). + +### 6. CLI machine: match cards and assess results {#initial-inventory-match} + +Launch Kit compares physical card serials returned by the selected sources. Each explicit DPU must match exactly one adapter on one observed host. A missing serial, conflicting identity or multiple matches prevents a complete inventory; an IP address or PCI address cannot substitute for that match. With host-only input, the inventory lists host-reported BlueFields without claiming a working ARM or BMC connection. These records remain partial and fail the required DPU BMC administrator check; ARM SSH cannot bypass that requirement. + +Next, the CLI compares eligible measured versions with the selected release's documented requirements. It reports unavailable measurements and unknown requirements separately from below-minimum versions. These comparisons do not install or update software. + +If address lookup is configured, the CLI separately matches your supplied management MACs to local lease-file records and neighbors read from the selected SSH endpoints. These are unverified IP candidates, not card identities or host-to-DPU associations. No candidate is contacted automatically. + +Finally, the CLI checks collection and session-cleanup results, compares against any previous inventory, and writes the YAML and report. Required failures produce partial output; optional gaps remain recorded even when inventory completes. All output files are written on the CLI machine. + +Next: [Review saved results](#initial-inventory-inspect). + +### 7. CLI machine: review inventory and reasons {#initial-inventory-inspect} + +The normal text summary shows the required-check result and named host/DPU mappings. It separates discovery blockers from later provisioning prerequisites and checks not requested. Repeated qualification/BOM gaps are grouped by affected sources; they do not mean a hardware failure was detected. Internal identity/comparison SHA identifiers remain in JSON/YAML, not as operator-facing device names. Both host and DPU results use named facts and field-specific issues; device evidence references, raw command strings and vendor firmware metadata are omitted. Saved YAML omits evidence maps and references from every source, including management, SSH, address lookup and connectivity probes. Diagnostic JSON is separate and can retain internal bookkeeping. + +Inspect `discovery.outcome`, `discovery.checks`, `errors`, `artifacts` and `acceptance` in the JSON report. Review saved `inventory.hosts`, `inventory.dpus`, `inventory.associations` and field-specific issues before using the handoff. Complete discovery writes `dpf-discovered.yaml`; partial discovery preserves an existing handoff and writes `partial.yaml` and `report.json` under `dpf-discovered.discovery//`. Use the report's artifact paths instead of guessing the run ID. + +Read each host/DPU `issues` entry's `field`, `code`, `message` and `tool` where present. Successful values live in named fields such as `psid`, `bmcAdministratorVerified`, `runtime.mftVersion` and `runtime.hardware.cards[].firmwareVersion`. Cards also retain typed `nextBoot` settings and `hostPrivileges`; next-boot values do not prove the running mode. A public `id` requires successful final identity rechecks; a serial retained on a failed record is diagnostic context. Generic observation maps and repeated successful state are omitted. + +DPU `firmware` groups version lists by component, retaining known activation separately. `firmwareInventoryComplete: true` means enumeration completed, not that every version is known or compatible. ERoT firmware remains separate from BMC firmware. The JSON report retains all required check outcomes and deferred provisioning checks. New inventories use `v1alpha3`; the updated CLI also reads legacy `v1alpha1`/`v1alpha2` without inferring missing verification. + +A missing `mlxfwmanager` is an unavailable prerequisite, not proof of incompatible NIC firmware. A denied Redfish firmware read differs from a property that is not exposed. Conflicting versions or unknown activation cannot establish a software minimum. + +| Example reason | What to do | +| --- | --- | +| `TOOL_NOT_AVAILABLE` / `PERMISSION_DENIED` | Check the named endpoint tool or authorized noninteractive sudo policy; do not treat the unmeasured check as passed. | +| `ACCESS_DENIED` / `PROPERTY_NOT_EXPOSED` | Distinguish an account permission problem from a property the service does not advertise. | +| `SOFTWARE_VERSION_CONFLICT` / `SOFTWARE_IDENTITY_UNAVAILABLE` | Reconcile version/activation sources or the rejected physical identity before trusting software comparisons. | +| `SOFTWARE_BELOW_MINIMUM` | Review the release requirement and plan a separate maintenance action; discovery does not update firmware. | +| `HOST_ADAPTER_VENDOR_CONFLICT` | Inspect the selected endpoints and matching serial/PCI evidence: a same-serial host adapter reports a non-NVIDIA vendor. Correct the source and rediscover; do not edit observed identity or discard a contradictory candidate to force a match. | +| `BMC_ADMIN_REQUIRED` / `BMC_ADMIN_ACCESS_DENIED` | Use an authorized account with all standard Administrator privileges and readable account/role evidence. | +| `BMC_ADMIN_UNVERIFIABLE` / `BMC_ADMIN_NOT_VERIFIED` | Supply explicit trusted DPU BMC access and inspect session Roles, authentication origin and linked account/role evidence. Host/ARM SSH and a same-named local administrator cannot substitute for verified session privileges. | +| `BMC_PASSWORD_CHANGE_REQUIRED` / `BMC_ADMIN_ACCOUNT_RESTRICTED` | Ask the administrator to resolve password-change, disabled or locked state separately, then rerun. Discovery does not repair access. | +| `PUBLICATION_FAILED` | Read the error explanation, resolve the local output/path problem, and rerun; only listed successfully written artifacts exist. | + +The resolved software BOM is saved under `inventory.software`: operator/component coordinates, exact hardware models and BFB version/URL/known digest, with documented minima in named fields such as `minimumNICFirmware` and `minimumHostMFT`. Missing minima remain unknown; catalog sources/revisions and generic requirement lists are omitted. A missing artifact digest is not fabricated. Release `26.4.0` has exact BlueField-3 model entries `B3210`, `B3220` and `B3240`; generic family names and BlueField-4 are not sufficient for BOM applicability. The documented NIC/MFT minima can be evaluated when current measurements associated with the correct device exist. BMC, CEC, ATF, UEFI, OS/BSP and golden-image minima remain unknown where the catalog has no supported requirement. A BFB target is a desired artifact, not an observed OS/BSP version. An unknown requirement must remain unresolved; supplying a guessed minimum does not establish compatibility. + +Exit zero and `discovery.outcome: complete` mean inventory collection, identity/mapping, cleanup and publication succeeded. `acceptance: not-evaluated` remains explicit. Hardware support, DPF authentication, service connectivity and deployment still need separate assessment; DPF generate/deploy/validate are unavailable in this preview. + +Next: [Complete inventory reviewed](#initial-inventory-finish); [Required failure or conflict](#initial-inventory-resolve). + +### 8. Endpoint owner or CLI operator: correct the cause {#initial-inventory-resolve} + +Use the specific recorded reason to correct the source configuration or ask the infrastructure owner to repair missing tools, permissions, credentials, connectivity or conflicting identity. Do not delete unknown observations or label them passed. Optional gaps may remain documented without blocking inventory; required source failures, identity ambiguity, cleanup or publication errors need resolution. If cleanup is uncertain, use the reported session/resource ownership evidence before any manual action. Rerun after correcting the cause. + +Next: [Retry after correction](#initial-inventory-prepare). + +### 9. CLI machine: retain the inventory and report {#initial-inventory-finish} + +Retain the seed, discovered YAML, JSON report and stderr log together. The discovered YAML preserves configuration intent and replaces observed inventory; it is not a deployment-ready profile or a provisioning approval. If a DPU access address is missing, use [address-candidate lookup](#address-candidates). For later remeasurement of the same estate, use [refresh](#refresh-inventory). + +## Find and verify address candidates {#address-candidates} + +Use this optional inventory addition when you know an ARM or BMC management MAC but need its possible IP address. It is a locator workflow, not authentication or identity discovery. You still need the host configuration from the first workflow; this is not a standalone subnet scanner. + +```mermaid +flowchart TD + n1["CLI machine: prepare MACs and lease snapshots"] + n2["CLI machine: configure address lookup"] + n3["CLI machine: run inventory and local lookup"] + n4["CLI operator: review candidate addresses"] + n5["Endpoint owner and CLI operator: verify and select"] + n1 -->|"MAC and source selected"| n2 + n2 -->|"Selectors configured"| n3 + n3 -->|"Candidate report available"| n4 + n4 -->|"Ownership can be independently verified"| n5 + n4 -->|"Missing or ambiguous evidence"| n1 +``` + +### 1. CLI machine: prepare MACs and lease snapshots {#address-candidates-locators} + +Obtain the exact ARM or BMC management MAC from authoritative asset records or a trustworthy observed interface. You must choose its role (`arm` or `bmc`) and a unique name. Launch Kit does not infer a management MAC from a NIC serial, data-port MAC, hostname or neighboring address. + +Optionally obtain a current dnsmasq IPv4 lease-file snapshot through your authorized DHCP administration process and place it on the CLI machine. Launch Kit reads this file locally; it does not log in to a DHCP server or query arbitrary vendor lease APIs. With no lease files, it can use suitable neighbors observed on the SSH endpoints already selected for inventory. Only `REACHABLE` or `PERMANENT` neighbors are candidates; stale entries are excluded. + +Next: [MAC and source selected](#address-candidates-configure). + +### 2. CLI machine: configure address lookup {#address-candidates-configure} + +Merge this fragment into the existing `discovery` section of a seed without saved inventory. Replace the MACs and file path; omit `sources` if you only want eligible observed neighbors. Do not replace the required host list. + +```yaml +discovery: + addressLookup: + sources: + - file: ./inputs/dnsmasq.leases + format: dnsmasq-leases + devices: + - name: dpu-a-arm + role: arm + mac: '02:00:00:00:00:11' + - name: dpu-a-bmc + role: bmc + mac: '02:00:00:00:00:12' +``` + +The supported file contains the five dnsmasq IPv4 columns: expiry epoch, MAC, IP, hostname and client ID. An expiry of `0` means a nonexpiring record; expired records are not candidates. Each file must be readable, at most 1 MiB and at most 4096 records. A malformed file contributes no candidates and retains a source failure reason. Other DHCP formats are unsupported. + +Next: [Selectors configured](#address-candidates-run). + +### 3. CLI machine: run inventory and local lookup {#address-candidates-run} + +Save the configured seed as `dpf-lookup.yaml` and select a new handoff path. + +```bash +l8k discover --target dpf --user-config dpf-lookup.yaml \ + --save-cluster-config dpf-candidates.yaml --output json \ + > candidates.json 2> candidates.log +``` + +The selected endpoints undergo the same [identity and optional reads](#initial-inventory-collect) as first inventory. On the CLI machine, address lookup then reads the configured lease snapshots and matches the supplied MACs against those records and the neighbors returned by remote `ip -j neighbor show`. No candidate is contacted and no credential is sent to it. + +Watch live source/check status on stderr: RUNNING is emitted before the operation, followed by its result. Endpoint names/addresses distinguish concurrent work. To keep progress visible with JSON, redirect stdout only (`>discover.json`). Add `--log-level debug` for stage timing or `--log-level trace` for request/tool metadata; `2>discover.log` saves these messages instead of displaying them. Quiet suppresses progress. + +Next: [Candidate report available](#address-candidates-review). + +### 4. CLI operator: review candidate addresses {#address-candidates-review} + +Read `inventory.addressLookup.candidates` for the address, selected name/role/MAC and source. Read `inventory.addressLookup.observations` for each device and lease source. A single address is `ADDRESS_UNVERIFIED`; several addresses for a MAC remain ambiguous, and missing/expired/unreadable evidence keeps its reason. A candidate is not a host-to-DPU association and is never a readiness pass. + +Use your DHCP/network or asset owner to resolve multiple, missing or contradictory locators. Do not choose an address merely because it is the first result. Independent identity and endpoint trust are still required even for one candidate. + +Next: [Ownership can be independently verified](#address-candidates-bind); [Missing or ambiguous evidence](#address-candidates-locators). + +### 5. Endpoint owner and CLI operator: verify and select {#address-candidates-bind} + +Verify endpoint ownership and obtain SSH known-hosts trust or Redfish CA trust and the appropriate account through your normal process. Create a new seed and add the verified endpoint under `discovery.dpus`, with its credential reference and trust policy. Use a new output when changing scope; do not edit the old observed identity to force a match. + +Run [initial inventory](#initial-inventory) for that explicit endpoint. Only its authenticated physical-card observation and unique host-card association can establish the binding. A discovered IP alone never authorizes Redfish access or proves Zero Trust attestation. + +## Refresh an existing inventory {#refresh-inventory} + +Use this workflow to remeasure the same selected estate, compare facts and preserve diagnostic history. A refresh is another live read; old successful observations do not stand in for unavailable current measurements. + +```mermaid +flowchart TD + n1["CLI machine: check inputs and trust before refresh"] + n2["CLI machine: repeat reads on the selected endpoints"] + n3["CLI machine: compare new observations and reasons"] + n4["Endpoint owner or CLI operator: repair and retry"] + n5["CLI machine: start a new inventory"] + n6["CLI machine: retain the refreshed files"] + n1 -->|"Same scope ready"| n2 + n1 -->|"Scope changed"| n5 + n2 -->|"Refresh finished"| n3 + n3 -->|"Complete refresh reviewed"| n6 + n3 -->|"Partial or failed refresh"| n4 + n4 -->|"Same scope corrected"| n2 + n4 -->|"Actual scope change"| n5 +``` + +### 1. CLI machine: check inputs and trust before refresh {#refresh-inventory-scope} + +Start from the existing discovered YAML and keep its credential/trust references valid. Decide whether you are remeasuring the same host/DPU endpoints and trust policy. If endpoint selection, trust policy or physical identity has changed, create a fresh input without old `inventory` and use a new output path. Investigate unexpected replacement or ownership changes before accepting them. + +The saved `scopeDigest` guards resolved endpoint/trust selection; the inventory does not repeat the configured policy list as `requestedScope`. Removing private device provenance alone is not a physical change. + +Changing Redfish certificate verification or the contents of a custom CA file requires a new inventory. Moving an unchanged CA file to another path does not. Keep the CLI machine's operating-system trust store maintained; discovery does not report changes within that store. + +The CLI reads legacy `v1alpha1` and `v1alpha2` inventories conservatively, but some older identity/trust bindings cannot support safe refresh. If the command reports that the old inventory is incompatible, verify endpoint ownership, create a fresh input without saved `inventory`, and use a new output path. Keep the previous file as history; do not edit its recorded identities or trust information to bypass the error. + +Do not manually edit observed serials, associations, firmware or check outcomes. `dpf.release` selects the release's BOM; changing it does not upgrade software. An omitted release uses the CLI's default, so record the CLI version and review `inventory.software` when comparing runs from different versions. Specify the release explicitly when you want to retain that choice. + +Next: [Same scope ready](#refresh-inventory-run); [Scope changed](#refresh-inventory-new-scope). + +### 2. CLI machine: repeat reads on the selected endpoints {#refresh-inventory-run} + +Launch Kit repeats the [same reads on the selected host, DPU, BMC and optional management API](#initial-inventory-collect), then matches the returned identities and compares them with the saved inventory. Run from the CLI machine using the same YAML path for input and output and explicitly supply `--save-cluster-config`: + +```bash +l8k discover --target dpf --user-config dpf-discovered.yaml \ + --save-cluster-config dpf-discovered.yaml --output json \ + > refresh.json 2> refresh.log +``` + +A complete in-place refresh retains `source-before.yaml` before replacing inventory while preserving user sections, comments and input intent. A partial run leaves the original input unchanged and writes its new observations separately; it does not create this backup. Relative file references are rebased if saving elsewhere; private files are not copied. Avoid simultaneous external edits, and use real output directories rather than symlink aliases. Credentials, trust files and configured lease files cannot be used as output destinations. + +Watch live source/check status on stderr: RUNNING is emitted before the operation, followed by its result. Endpoint names/addresses distinguish concurrent work. To keep progress visible with JSON, redirect stdout only (`>discover.json`). Add `--log-level debug` for stage timing or `--log-level trace` for request/tool metadata; `2>discover.log` saves these messages instead of displaying them. Quiet suppresses progress. + +Next: [Refresh finished](#refresh-inventory-compare). + +### 3. CLI machine: compare new observations and reasons {#refresh-inventory-compare} + +Review `diff`, current checks, `errors`, cleanup and successfully written `artifacts` in the result. Compare physical identity, firmware, relevant hardware/mode/network observations and resolved release requirements. A failed current observation stays unknown rather than inheriting an earlier pass. Reordered lists alone are not changes to the observed configuration. + +On partial collection, the previous handoff remains intact and the new partial YAML/report live under `dpf-discovered.discovery//`. Inspect the new report instead of mistaking the unchanged handoff for fresh success. Required failures need correction and another run. Optional missing tools or inaccessible properties remain explained observations; they do not make deployment readiness pass. + +Next: [Complete refresh reviewed](#refresh-inventory-retain); [Partial or failed refresh](#refresh-inventory-repair). + +### 4. Endpoint owner or CLI operator: repair and retry {#refresh-inventory-repair} + +Use the saved reason and affected source to repair the external prerequisite or fix the input. Preserve the unsuccessful run's report. Do not overwrite failure evidence with an old successful observation. If the cause is a real endpoint, trust or identity change, switch to a fresh seed and output. + +Next: [Same scope corrected](#refresh-inventory-run); [Actual scope change](#refresh-inventory-new-scope). + +### 5. CLI machine: start a new inventory {#refresh-inventory-new-scope} + +Create a new seed from the desired access configuration, without copying old `inventory` facts, and select a new handoff path. Recheck endpoint ownership/trust and follow [initial inventory](#initial-inventory). Keep the previous estate's handoff and report as history. + +### 6. CLI machine: retain the refreshed files {#refresh-inventory-retain} + +Keep the current YAML, JSON/stdout report and stderr log, plus `source-before.yaml` when a complete in-place refresh created it. Discovery completion still means a successful inventory workflow, with deployment `acceptance: not-evaluated`. Unknown software requirements and unverified deployment prerequisites remain unresolved; a refreshed inventory does not enable DPF deployment. diff --git a/docs/user/heterogeneous-clusters.md b/docs/user/heterogeneous-clusters.md index ac10197a..50fc2bfd 100644 --- a/docs/user/heterogeneous-clusters.md +++ b/docs/user/heterogeneous-clusters.md @@ -16,15 +16,11 @@ Each `clusterConfig` entry is a source group with its own: - East-west PF inventory, rail assignments, and capabilities. - Storage and third-party RDMA kernel modules. -Fresh discovery derives `identifier` from the machine/GPU identity, lowercases -it, and removes complete `NVIDIA` segments. For example, -`DGX-B200-NVIDIA-H200` becomes `dgx-b200-h200`; the Launch Kit machine node -label uses the same value. Common machine segments are shortened in the same -pass (`ThinkSystem` → `ts`, `PowerEdge` → `pe`). Identifiers are bounded to 30 -bytes. Long values use balanced machine/GPU prefixes plus a 6-character -deterministic hash, with unused prefix space reassigned to the longer component. -The separate GPU label still retains its discovered value such as `NVIDIA-H200`. -Use the persisted identifier shown in `cluster-config.yaml` with `--groups`. +Discovery saves a lowercase group identifier and uses the same value for the +Launch Kit machine label. Names may be shortened to fit the identifier limit. +The separate GPU label retains a value such as `NVIDIA-H200`. Use the saved +identifier from `cluster-config.yaml` with `--groups`; do not construct it from +the machine and GPU names yourself. Inspect the available groups before filtering: diff --git a/docs/user/installation.md b/docs/user/installation.md index 3d98a150..2b30cbf9 100644 --- a/docs/user/installation.md +++ b/docs/user/installation.md @@ -11,10 +11,9 @@ embedded release catalog; use `l8k schema` to see the release lines that **your installed binary** recognizes. Complete [deployment planning](profiles.md) before using cluster credentials. -This website is published from the repository's `main` branch. For a pinned -binary, read the documentation at its repository release tag when you need the -exact behavior of that version. Record the `l8k version` output and the -selected Network Operator release with your deployment artifacts. +Use documentation matching your installed Launch Kit release. Record the +`l8k version` output and selected Network Operator release with your deployment +artifacts. ## Prerequisites @@ -79,28 +78,6 @@ l8k() { The generated configuration, manifests, and reports remain in the current directory. Ensure the Kubernetes API server address in the mounted kubeconfig is reachable from the container runtime. -## Build From Source - -```bash -git clone https://github.com/NVIDIA/k8s-launch-kit.git -cd k8s-launch-kit -make build -``` - -The binary is written to `build/l8k`. - -For a local install: - -```bash -make install -``` - -For development, symlink the source-tree assets instead of copying them: - -```bash -make dev-install -``` - ## Installed Assets The install places the binary on PATH. The default configuration and bundled topology presets are embedded in the binary. Use `--config-dir` when you need a filesystem override: @@ -132,10 +109,8 @@ Build-only users should run from the repository root or install the profiles. Release artifacts also contain the Network Operator sosreport helper. The installers place it at -`/share/l8k/scripts/kubectl-netop_sosreport`. Source-tree -`make install`, `make dev-install`, and direct `scripts/install-local.sh` -invocations download the helper when it is absent; `l8k sosreport` itself never -downloads executable code at runtime. +`/share/l8k/scripts/kubectl-netop_sosreport`. Keep this helper installed +alongside the binary; `l8k sosreport` does not download executable code at runtime. ## Verify diff --git a/docs/user/openshift.md b/docs/user/openshift.md index c2d428a0..555336d1 100644 --- a/docs/user/openshift.md +++ b/docs/user/openshift.md @@ -5,7 +5,7 @@ SPDX-License-Identifier: Apache-2.0 # OpenShift host deployments -This procedure prepares **SR-IOV Ethernet RDMA** on an existing OpenShift cluster. Ethernet SR-IOV was exercised on a two-node OpenShift 4.22 cluster. Other standard OpenShift profiles passed rendering and server dry-run but still need live integration qualification. Spectrum-X has no OpenShift variant. The example uses Network Operator release `26.7`; confirm its suitability for the site before proceeding. +This procedure prepares **SR-IOV Ethernet RDMA** on an existing OpenShift cluster with externally installed operators. Other standard OpenShift profiles are preview workflows: confirm runtime support for the selected hardware, operator release and profile before use. Spectrum-X has no OpenShift variant. The example uses Network Operator release `26.7`; choose an operator release compatible with your OpenShift version and hardware. ## Before you start diff --git a/docs/user/presets.md b/docs/user/presets.md index e9839762..fea3c119 100644 --- a/docs/user/presets.md +++ b/docs/user/presets.md @@ -151,9 +151,8 @@ false or omitted. The override replaces the catalog, so include every preset needed for the review. Inspect networks, resource names, selectors, and GPU requests in the rendered output. Rendering does not certify the hardware. -5. Verify the complete PF inventory against representative live discovery and - inspect any preset deviations. Submit the preset and its evidence through - the repository contribution process. +5. Compare the complete PF inventory with discovery from the target hardware + and resolve any preset deviations before using the preset for deployment. ## Validation And Deviations diff --git a/docs/user/profiles.md b/docs/user/profiles.md index 63dddcb4..e24cf03d 100644 --- a/docs/user/profiles.md +++ b/docs/user/profiles.md @@ -31,20 +31,20 @@ resource requests, and workload namespaces. Launch Kit resolves those inputs, renders resources, and reports its observed deployment checks. A site owner decides which checks and coverage constitute acceptance for the workload. -## Availability and evidence +## Workflow availability -| Workflow | Documented implementation | Recorded verification | +| Workflow | Use and prerequisites | Limits | | --- | --- | --- | -| Kubernetes standard profiles below | Generation, deployment, and validation paths are documented. | Consult the relevant release/hardware qualification record; this guide does not establish live qualification for every combination. | -| OpenShift SR-IOV Ethernet | OpenShift-specific generation, deployment, and validation. | Exercised on a two-node OpenShift 4.22 cluster. | -| Other OpenShift standard profiles | Rendering and server dry-run. | Live integration qualification on suitable hardware remains to be recorded. | -| OpenShift Spectrum-X | No generated OpenShift variant. | Unavailable; see [OpenShift qualification](openshift.md). | -| Kubernetes Spectrum-X RA2.1, RA2.2, RA2.3, RA2.4, RA3.1 | Release-specific profile generation as shown in the [version matrix](spectrum-x.md#version-matrix). | Confirm the exact hardware, RA, operator release, and validated site profile with the responsible owner. | - -Rendering, API dry-run, live deployment, and a formal support commitment are -different evidence. The table describes what this repository documents; it -does not extend qualification to an untested platform or hardware combination. -The reserved `dpf` target has no available lifecycle phases. +| Kubernetes standard profiles below | Select the fabric and workload device model that match your NICs and network. | Confirm the selected operator release and hardware combination with the platform owner before deployment. | +| OpenShift SR-IOV Ethernet | Follow the [OpenShift procedure](openshift.md) with the required operators already installed. | Launch Kit configures those operators; it does not install their Helm charts. | +| Other OpenShift standard profiles | Select an OpenShift-specific profile and review its generated resources with the operator owners. | Treat these profiles as preview workflows until runtime support for your hardware and release is confirmed. | +| OpenShift Spectrum-X | No OpenShift variant is available. | Use a supported Kubernetes Spectrum-X environment. | +| Kubernetes Spectrum-X RA2.1, RA2.2, RA2.3, RA2.4, RA3.1 | Choose a compatible release using the [version matrix](spectrum-x.md#version-matrix) and supply the site profile/topology. | Hardware, fabric and operator versions must match the selected RA requirements. | +| DPF inventory | Use the separate [DPF discovery workflow](dpf-workflows.md) to inventory host and DPU endpoints. | Experimental inventory only; DPF provisioning phases are unavailable. | + +Choose a release and hardware combination supported by your platform and +operator vendors. Successful generation or API dry-run alone does not establish +deployment readiness; review the deployment and workload validation results. ## Check prerequisites and site inputs diff --git a/docs/user/spectrum-x.md b/docs/user/spectrum-x.md index 0c9c4556..e2c79b3e 100644 --- a/docs/user/spectrum-x.md +++ b/docs/user/spectrum-x.md @@ -169,7 +169,7 @@ namespace always matches the resolved Network Operator namespace, including **Release prerequisite:** Network Operator v26.10.0-beta.2 includes the rail-pool `swPlane` and `ovsConfig` fields, but its packaged NCO CRD lacks `platformType`. RA2.4/RA3.1 deployment requires a 26.10-or-newer operator build that includes the -doSPCX NCO implementation and matching CRDs. Generation alone does not establish +doSPCX-capable NIC Configuration Operator and matching CRDs. Generation alone does not establish that the selected published chart satisfies this prerequisite. Deploy reads the live NCO CRD after NicClusterPolicy/NicNodePolicy bootstrap and requires the NIC template's served API version to expose `platformType` as a string. An absent diff --git a/docs/user/troubleshooting.md b/docs/user/troubleshooting.md index dea0f38c..6471bf1f 100644 --- a/docs/user/troubleshooting.md +++ b/docs/user/troubleshooting.md @@ -179,8 +179,7 @@ archives contain the collection script, and supported installers place it at `/share/l8k/scripts/kubectl-netop_sosreport`. The command does not download executable code at runtime. If the script is missing, its error reports the raw source URL and exact expected path so an executable copy -can be installed manually. Source-tree developers can populate their checkout -with `make download-sosreport`. +can be installed manually. Start with the diagnostic summary and collection errors, then correlate: diff --git a/docs/user/validation.md b/docs/user/validation.md index a34c49ab..79197618 100644 --- a/docs/user/validation.md +++ b/docs/user/validation.md @@ -208,9 +208,7 @@ The root is ``. Each family is a direct `` child with `tests`, `failures`, `errors`, `skipped` and `time` attributes. Each directional pod/rail probe is a direct `` child, named `FamilyName::source→destination`; endpoint names include node, rail, pod -and IP to distinguish probes. There is no extra aggregate testcase. Counters -count the child cases once. This structure is readable by ai-cloud-validation's -JUnit parser; nested `` elements would hide the individual results. +and IP to distinguish probes. Suite counters summarize these individual results. Suite time is measured wall-clock seconds for the family stage, including route checks. Individual probe times are omitted because batched RDMA execution does diff --git a/engineering/README.md b/engineering/README.md new file mode 100644 index 00000000..65b327f8 --- /dev/null +++ b/engineering/README.md @@ -0,0 +1,38 @@ + + +# Engineering documentation + +These documents support Launch Kit development. They are not published in the +operator documentation or included in user documentation exports. + +- [Current architecture and Go library](architecture/overview.md) +- [Read-only discovery library](architecture/discovery-library.md) +- [Artifact bundle internals](architecture/artifact-bundle.md) +- [Target extension contract](advanced/targets.md) +- [DPF integration plan](architecture/dpf-integration-plan.md) +- [Host target migration history](architecture/host-target-migration-plan.md) +- [Documentation maintenance and publishing](reference/docs-publishing.md) + +See [CONTRIBUTING](../CONTRIBUTING.md) for development and review requirements. +Operator procedures and reference material belong under `docs/`; design plans, +implementation details and development validation belong here. Lab-specific +credentials, cluster inventories and test-run artifacts stay outside the repository. + +## Build From Source + +```bash +git clone https://github.com/NVIDIA/k8s-launch-kit.git +cd k8s-launch-kit +make build +``` + +The binary is written to `build/l8k`. + +For a local install: + +```bash +make install +``` diff --git a/docs/advanced/targets.md b/engineering/advanced/targets.md similarity index 88% rename from docs/advanced/targets.md rename to engineering/advanced/targets.md index ae645bec..79c32390 100644 --- a/docs/advanced/targets.md +++ b/engineering/advanced/targets.md @@ -29,15 +29,7 @@ Passing `--target host` is an explicit synonym: l8k generate --target host ... ``` -The `dpf` target name is reserved for the future DPU-plane workflow. Its -phases are intentionally reported as unavailable until a typed DPF driver is -installed. Launch Kit returns validation exit code `2` instead of falling -through to host behavior: - -```console -$ l8k discover --target dpf -Error: invalid target invocation: target "dpf" does not implement phase "discover" ... -``` +The `dpf` target implements an [experimental inventory discovery preview](../../docs/user/dpf-discovery.md). Its software collects explicit Redfish or host/ARM SSH inventory, optionally with management Kubernetes observations. Inventory acceptance is independent of DPF provisioning readiness; hardware qualification and management probe-image release gates remain explicit. Other DPF phases remain unavailable and cannot fall through to Host execution. Use `l8k schema` to inspect target and phase availability programmatically. @@ -103,8 +95,7 @@ binds through the registry and calls `Operation.Run(cmd.Context())` exactly once. Discover, generate, and the root pipeline reuse the Host application launcher; standalone deploy and validate are Host-owned services outside Cobra. Those services return errors, while `pkg/cmd` remains the only process -termination boundary. The DPF target remains unavailable, so it cannot -accidentally construct a Kubernetes client or execute Host logic. +termination boundary. The DPF discovery branch binds its own typed request before collecting Host flags, and constructs only DPF dependencies. ## Registry guarantees @@ -132,7 +123,7 @@ A target integration should: domain driver. 5. Declare all four phase capabilities and their required logical cluster roles. -6. Implement all four phases before marking them available in the schema. +6. Mark only implemented phases available; document preview and qualification limits. 7. Keep target artifacts separate; host output remains `deployment/network-operator/`. 8. Add omitted-target and explicit-host compatibility tests before routing the diff --git a/docs/architecture/artifact-bundle.md b/engineering/architecture/artifact-bundle.md similarity index 100% rename from docs/architecture/artifact-bundle.md rename to engineering/architecture/artifact-bundle.md diff --git a/engineering/architecture/discovery-library.md b/engineering/architecture/discovery-library.md new file mode 100644 index 00000000..22100053 --- /dev/null +++ b/engineering/architecture/discovery-library.md @@ -0,0 +1,95 @@ + + +# Discovery library + +This reference is for Go library consumers, not CLI operators. + +## Read-only library discovery + +Library consumers that only need the inventory from an already-running NIC +Configuration Daemon can call `networkoperatorplugin.DiscoverReadOnly`. The +function reads +the existing daemon pods, nodes, and `NicDevice` resources and returns a +`LaunchKitConfig` without creating, patching, or deleting Kubernetes resources. +It locates the existing daemon by its DaemonSet owner in the configured +Network Operator namespace when `WithReadOnlyNetworkOperatorNamespace` is +provided. Without that option, it checks the standard Network Operator and +Launch Kit namespaces, preferring a usable Network Operator daemon when both +installations are present. +It returns `networkoperatorplugin.ErrNotInstalled` when the discovery daemon is +not already running. With the configured-namespace option, callers need pod +read access only in that namespace; otherwise they need pod-list access in all +fallback namespaces. Both paths need cluster-scoped read access to nodes and +all-namespaces read permission for the namespaced `NicDevice` resources. Use +`discovery.Discover` when the caller is responsible for bootstrapping the +temporary daemon. + +The function does not use node labels to guess whether a daemon node can +publish a `NicDevice`. If every Ready daemon node already has a device, no pod +execution is needed. If any device is missing, pass +`networkoperatorplugin.WithReadOnlyRESTConfig` with the REST config for the +cluster. The function then runs the existing read-only NIC and BlueField trust +probe in that node's daemon pod. It excludes nodes with no discoverable NIC or +only restricted BlueFields, and reports a missing device for a verified NIC +publisher. It returns inspection and permission errors instead of treating +them as absent hardware. This fallback needs `pods/exec` permission in the +daemon namespace. + +Without a REST config, discovery returns an error by default. A caller that +accepts an incomplete inventory can opt in with +`networkoperatorplugin.WithReadOnlyAssumeUnverifiedExcluded()`. Ready daemon +nodes without a `NicDevice` are then excluded and a warning reports that their +publisher status is unverified. This option does not suppress errors from a +probe attempted with a REST config. At least one `NicDevice` on a Ready daemon +node is still required to construct an inventory. + +For optional fabric enrichment, pass both +`networkoperatorplugin.WithReadOnlyRESTConfig(restConfig)` and +`networkoperatorplugin.WithReadOnlyFabricDiscovery()`. This explicitly permits +read-only execs in the existing daemon pods and needs `pods/exec` permission in +the selected daemon namespace. Supplying only a REST config retains the existing +publisher-verification behavior and does not enable fabric probes. The fabric +option without a REST config fails before cluster reads. + +Fabric enrichment checks every selected worker and east-west PF, including sibling +PFs represented by a collapsed rail. One batched command per worker confirms +PCI-local RDMA names, enumerates all ports and reads their configured `link_layer`. +Only complete, unanimous observations set a group's `linkType`; partial, denied, +unreadable or conflicting observations leave it unknown with diagnostics. Link +state and subnet-manager presence do not gate fabric detection. An InfiniBand PF +without a netdev gets a separate IPoIB host-provisioning warning. + +Normal discovery uses the same all-worker fabric check. Collection allows four +concurrent workers with a ten-second deadline per worker and no timeout retry. +Its optional worst-case budget is approximately `ceil(selected workers / 4) * 10s` +plus API/dispatch overhead, capped by the caller's deadline. Internal probe +failures leave fabric unknown; caller cancellation or deadline returns an error. +Neither optional enrichment nor partial-inventory opt-in suppresses publisher +inspection errors when REST credentials are supplied. These observations prove +coverage of a successfully enumerated sysfs snapshot, not absent kernel hardware +or deployment readiness. No host modules, node labels or Kubernetes resources are +changed by read-only discovery. + +GPU product/topology enrichment is separately opt-in: +`networkoperatorplugin.WithReadOnlyGPUTopology("gpu-operator")` requires the +actual GPU Operator namespace and `WithReadOnlyRESTConfig(restConfig)`. +Default calls and REST-only calls do not acquire GPU probes. This option reads +host PCI identities in existing NIC daemon pods and tries `nvidia-smi` there, +then uses a verified same-node GPU Operator `gpu-feature-discovery` container +when necessary. It does not run the netplan workflow or modify either daemon. +Callers need pod read/exec access in both namespaces and DaemonSet read access +in the GPU Operator namespace. Missing namespace or REST credentials fails +before cluster reads. Optional failures leave unavailable GPU fields unset; +caller cancellation remains an error. Supply and consume a `pkg/ui` context +sink to receive these warnings; the library's default sink is silent. + +By default, read-only discovery also fails if it cannot derive a node selector +that matches only a group's workers. Inventory-only callers can pass +`networkoperatorplugin.WithReadOnlyAllowMissingNodeSelector()` to return the +topology in that case. The affected group's `NodeSelector` is left empty and +`WorkerNodes` is authoritative; do not use that empty selector for workload +placement. This option does not relax errors for missing or unverified +`NicDevice` resources. diff --git a/docs/architecture/dpf-integration-plan.md b/engineering/architecture/dpf-integration-plan.md similarity index 84% rename from docs/architecture/dpf-integration-plan.md rename to engineering/architecture/dpf-integration-plan.md index 276d4441..99fd01a6 100644 --- a/docs/architecture/dpf-integration-plan.md +++ b/engineering/architecture/dpf-integration-plan.md @@ -317,7 +317,54 @@ Start with Zero Trust passthrough: - report the provenance and resolution state of every required value. Exit criteria: `l8k discover --target dpf` produces a complete or explicitly -unresolved DPF configuration and never guesses infrastructure values. +unresolved DPF configuration and never guesses infrastructure values. Completed +inventory requires read-only verification of the authenticated DPU BMC account and +its effective Administrator privileges. Host/ARM-only inventory is partial and +cannot bypass this gate. Preserve physical identity and authorization failure +evidence independently; never rotate passwords or probe privilege by mutation. +The current DPF target still rejects generation, deployment and validation before +consuming inventory. A future generator must require fresh complete discovery and +revalidate its own authorization needs; saved facts do not authorize provisioning. + +Inventory `v1alpha3` has a versioned public projection for both host and DPU +records; internal collector/check APIs keep their existing domain state. Public +facts use dedicated fields: identity, PSID, administrator verification, runtime +versions, per-card firmware/PSID, next-boot settings and host privileges. Named +DPU firmware components retain version/activation; raw vendor metadata and +command output are not serialized. `issues` carries only failures, mismatches +and unavailable facts, with field/code/message/tool rather than a generic +state/value wrapper. ID is exported only for an eligible final identity. +Version-aware decoding preserves these authority boundaries and rejects +contradictions. Legacy `v1alpha1` and `v1alpha2` remain readable without inferring +missing success. Required report check results and qualification boundaries stay +separate from compact device facts. Firmware conflict comparison follows the same +whitespace handling as collection while retaining original measured strings; +serialization does not make a padded version eligible for minimum-version checks. + +SSH collection keeps one connection for normal reads. A command deadline closes +that transport to bound stalled session operations; the next independent read may +reconnect with the original endpoint, credentials and trust snapshot. The failed +command is not replayed. Reconnection failure remains a required-identity failure, +and overall cancellation forbids recovery. Final physical identity rereads remain +mandatory after optional observations. + +The public software BOM is a projection of the internal release catalog: +operator/component coordinates, exact hardware applicability, BFB version/URL +and known digest, plus fixed named minimum fields. Missing minima reconstruct +unknown requirements for compatibility checks; decoding uses saved coordinates +rather than current catalog defaults. Catalog sources/revisions, generic +requirements and static limitation prose stay internal. Host BMC collection is +limited to identity, PCI mapping and advertised reset capabilities; DPU ERoT and +BMC firmware remain distinct components. + +Host/DPU records do not create evidence tables or reference lists, regardless of +transport. Mandatory live reads, identity and PSID rereads, administrator checks, +cleanup and publication remain unchanged. Management/probe evidence retains its +own contract. The public `scopeDigest` guards resolved refresh endpoints/trust; +`requestedScope` and its detailed policy bookkeeping stay internal. Material +fingerprints compare the public fact projection so removed provenance does not +create false changes. Refresh preserves source backups and existing stable +physical identity/scope conflict guards. ### Stage 4: DPF generation for Zero Trust passthrough diff --git a/docs/architecture/host-target-migration-plan.md b/engineering/architecture/host-target-migration-plan.md similarity index 99% rename from docs/architecture/host-target-migration-plan.md rename to engineering/architecture/host-target-migration-plan.md index fcbf3c18..8c2a3b03 100644 --- a/docs/architecture/host-target-migration-plan.md +++ b/engineering/architecture/host-target-migration-plan.md @@ -357,7 +357,7 @@ the process. The Cobra boundary remains the only owner of exit codes. The migration must pay particular attention to JSON mode: - preserve command-specific stdout contracts (including validation's JSON - stream); see [Automation](../integrator/automation.md#json-mode); + stream); see [Automation](../../docs/integrator/automation.md#json-mode); - human logs remain on stderr; - errors are finalized once, not by both `app.Launcher` and Cobra; - the existing Host JSON schema does not gain target fields during this work; diff --git a/docs/architecture/overview.md b/engineering/architecture/overview.md similarity index 98% rename from docs/architecture/overview.md rename to engineering/architecture/overview.md index 160cf2f4..1597dfd9 100644 --- a/docs/architecture/overview.md +++ b/engineering/architecture/overview.md @@ -49,7 +49,7 @@ flowchart TB validatesvc["Standalone validate + report service"] hostpaths["Host config, kubeconfig, preset, artifact resolution"] end - dpf["DPF target extension point\nrecognized; phases unavailable"] + dpf["DPF inventory — pkg/target/dpf + pkg/dpf\nexplicit Redfish / host SSH / ARM OOB SSH sources\nprovisioning phases unavailable"] end subgraph application["Host application orchestration"] @@ -397,7 +397,7 @@ point for applications that need Launch Kit's hardware inventory without the CLI. It bootstraps the discovery daemon, fills a `*config.LaunchKitConfig`, and writes the `nvidia.kubernetes-launch-kit.machine` and `.gpu` labels. It also patches an existing `NicClusterPolicy` when needed; the call changes cluster -state. See [Cluster Discovery](../user/discovery.md) for prerequisites and +state. See [Cluster Discovery](../../docs/user/discovery.md) for prerequisites and bootstrap effects. After creating a controller-runtime `client.Client` and matching diff --git a/docs/reference/docs-publishing.md b/engineering/reference/docs-publishing.md similarity index 81% rename from docs/reference/docs-publishing.md rename to engineering/reference/docs-publishing.md index 90242c8d..9fdc8e7b 100644 --- a/docs/reference/docs-publishing.md +++ b/engineering/reference/docs-publishing.md @@ -18,11 +18,10 @@ docs/ |-- integrator/ |-- reference/ |-- advanced/ -|-- architecture/ `-- assets/ ``` -The old RST files under `docs/` are retained for compatibility, but the GitHub Pages site uses the Markdown pages listed in `mkdocs.yml`. The DPF roadmap and HostTarget migration history are published and linked from [Architecture](../architecture/overview.md), outside the primary navigation. Keep their status labels and `llms.txt` entries current. +The old RST files under `docs/` are retained for compatibility. The published documentation uses the operator-facing Markdown pages listed in `mkdocs.yml`. Engineering architecture, implementation plans and contributor instructions live under `engineering/` and are excluded from user navigation, search indexes and HTML exports. Keep `docs/llms.txt` aligned with the published operator pages. ## Content Ownership @@ -36,8 +35,9 @@ contracts in README or skills. Task articles should state the outcome, applicability, inputs and permissions, ordered actions, expected evidence, and failure/next step. Reference articles -optimize lookup; architecture and design plans describe implementation or -future work. Label snippets that are illustrative or partial so readers do not +optimize lookup. Explain operations, where they execute, prerequisites, outputs +and recovery actions. Do not include package internals, implementation plans, +specific lab clusters or development test-run history in user documentation. Label snippets that are illustrative or partial so readers do not mistake them for a runnable, qualified site configuration. ## Local Build @@ -66,8 +66,7 @@ python3 scripts/check-docs.py --binary build/l8k The CI build job runs this check on every pull request and main-branch push. It compares long flags in Markdown/RST shell examples and Cobra help examples against the actual command flags, verifies the lifecycle applicability matrix -and the `llms.txt` inventory of every published Markdown page, including linked -design plans. It renders the published standard Profile YAML with an embedded +and the `llms.txt` inventory of every published Markdown page. It renders the published standard Profile YAML with an embedded preset and the complete application sample in an isolated temporary directory. It checks source preservation, effective metadata, operational workload namespace/resource transformation, and the fixture retention recipe. diff --git a/examples/dpf/discovery.yaml b/examples/dpf/discovery.yaml new file mode 100644 index 00000000..b47f6cba --- /dev/null +++ b/examples/dpf/discovery.yaml @@ -0,0 +1,25 @@ +# Copyright 2026 NVIDIA CORPORATION & AFFILIATES +# SPDX-License-Identifier: Apache-2.0 +# Synthetic site. Replace endpoints, kubeconfig and account references locally. +management: + kubeconfigRef: + file: ./credentials/management.kubeconfig + context: management +credentials: + dpu-reader: + username: bmc-reader + passwordRef: + env: DPU_BMC_PASSWORD + host-reader: + username: bmc-reader + passwordRef: + env: HOST_BMC_PASSWORD +discovery: + dpus: + - bmcEndpoint: https://192.0.2.11 + credentialRef: dpu-reader + hosts: + - bmcEndpoint: https://192.0.2.21 + credentialRef: host-reader + probes: + mode: inventory diff --git a/go.mod b/go.mod index b2ba9dcf..9389ddf8 100644 --- a/go.mod +++ b/go.mod @@ -8,6 +8,7 @@ require ( github.com/Mellanox/nic-configuration-operator v1.3.2-0.20261027094742-6806232e532b github.com/go-logr/logr v1.4.4 github.com/go-logr/zapr v1.3.0 + github.com/gofrs/flock v0.13.0 github.com/mitchellh/copystructure v1.2.0 github.com/onsi/ginkgo/v2 v2.34.0 github.com/onsi/gomega v1.44.0 @@ -15,6 +16,8 @@ require ( github.com/spf13/pflag v1.1.0 github.com/stretchr/testify v1.12.1 go.uber.org/zap v1.28.0 + golang.org/x/crypto v0.57.0 + golang.org/x/sys v0.49.0 golang.org/x/term v0.47.0 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 @@ -117,12 +120,10 @@ require ( go.uber.org/multierr v1.11.0 // indirect go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect - golang.org/x/crypto v0.57.0 // indirect golang.org/x/mod v0.41.0 // indirect golang.org/x/net v0.60.0 // indirect golang.org/x/oauth2 v0.36.0 // indirect golang.org/x/sync v0.24.0 // indirect - golang.org/x/sys v0.49.0 // indirect golang.org/x/text v0.42.0 // indirect golang.org/x/time v0.15.0 // indirect golang.org/x/tools v0.49.0 // indirect diff --git a/go.sum b/go.sum index ab649ec9..69ce4357 100644 --- a/go.sum +++ b/go.sum @@ -147,6 +147,8 @@ github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y= github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8= github.com/goccy/go-yaml v1.18.0 h1:8W7wMFS12Pcas7KU+VVkaiCng+kG8QiFeFwzFb+rwuw= github.com/goccy/go-yaml v1.18.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA= +github.com/gofrs/flock v0.13.0 h1:95JolYOvGMqeH31+FC7D2+uULf6mG61mEZ/A8dRYMzw= +github.com/gofrs/flock v0.13.0/go.mod h1:jxeyy9R1auM5S6JYDBhDt+E2TCo7DkratH4Pgi8P+Z0= github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg= github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4= github.com/google/gnostic-models v0.7.0 h1:qwTtogB15McXDaNqTZdzPJRHvaVJlAl+HVQnLmJEJxo= diff --git a/mkdocs.yml b/mkdocs.yml index 380c60b7..92177581 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -22,9 +22,6 @@ repo_url: https://github.com/NVIDIA/k8s-launch-kit edit_uri: edit/main/docs/ exclude_docs: | *.rst -not_in_nav: | - /architecture/dpf-integration-plan.md - /architecture/host-target-migration-plan.md theme: name: material @@ -90,6 +87,8 @@ nav: - Networking concepts: advanced/concepts.md - Deploy: - Discover cluster hardware: user/discovery.md + - DPF discovery workflows: user/dpf-workflows.md + - DPF discovery preview: user/dpf-discovery.md - Generate and review manifests: advanced/generation.md - Deploy manifests: advanced/deployment.md - Deploy on OpenShift: user/openshift.md @@ -108,11 +107,6 @@ nav: - Reference: - CLI reference: reference/cli.md - Configuration reference: reference/configuration.md - - Contribute: - - Architecture: architecture/overview.md - - Artifact bundle internals: architecture/artifact-bundle.md - - Extend deployment targets: advanced/targets.md - - Maintain and publish documentation: reference/docs-publishing.md validation: omitted_files: warn diff --git a/openspec/README.md b/openspec/README.md index f9882f69..ab8c2091 100644 --- a/openspec/README.md +++ b/openspec/README.md @@ -87,3 +87,19 @@ promise across several boundaries. It is optional for routine conforming changes these contracts introduce no mandatory proposal/approval stage or per-flag ceremony. Keep exact flag/default/version inventories in their existing mechanical sources, operating procedures in docs/skills, and development rules in AGENTS.md. + +## Procedural workflow documentation + +[Repository-owned workflow definitions](workflows/README.md) under `workflows/` +provide the single source for the DPF CLI procedures and their flow diagrams. +`python3 scripts/generate-workflow-docs.py` renders +[DPF workflows](../docs/user/dpf-workflows.md); `--check` detects drift without +writing files. The Docs CI job runs the generator tests and drift check before +building the site. + +This is a Launch Kit documentation extension colocated with OpenSpec contracts, +**not a native executable OpenSpec workflow format**. The generator does not run +commands, validate infrastructure or enforce runtime workflow transitions. OpenSpec +structural validation remains separate; neither check proves implementation +conformance. Procedures must be reviewed against the CLI and relevant contract +scenarios when behavior changes. diff --git a/openspec/specs/artifact-handoff/spec.md b/openspec/specs/artifact-handoff/spec.md index fa73cd2d..cad253c1 100644 --- a/openspec/specs/artifact-handoff/spec.md +++ b/openspec/specs/artifact-handoff/spec.md @@ -127,4 +127,4 @@ Structural validation does not prove CRD schema/admission or readiness. Sidecar - Implementation: [bundle](../../../pkg/bundle), [effective config](../../../pkg/config/effective.go), [Host paths](../../../pkg/target/host/paths.go). - Existing suites: [bundle](../../../pkg/bundle/bundle_test.go), [application bundle](../../../pkg/app/bundle_test.go), [Host bundle](../../../pkg/target/host/bundle_test.go), [effective config](../../../pkg/config/effective_test.go), [paths](../../../pkg/target/host/paths_test.go). -- Architecture: [artifact bundle](../../../docs/architecture/artifact-bundle.md). +- Architecture: [artifact bundle](../../../engineering/architecture/artifact-bundle.md). diff --git a/openspec/specs/configuration-parameters/spec.md b/openspec/specs/configuration-parameters/spec.md index bdd87852..d40d0b03 100644 --- a/openspec/specs/configuration-parameters/spec.md +++ b/openspec/specs/configuration-parameters/spec.md @@ -57,6 +57,20 @@ Resolution SHALL leave caller inputs and shared defaults unchanged. Extensions M - **WHEN** one caller modifies its resolved nested configuration - **THEN** a subsequent resolution and the original source/defaults retain their own values. +### Requirement: DPF software release defaults + +The DPF target SHALL resolve software coordinates from a release-owned BOM, separately from site access inputs and observed running versions. Omitted or null release selection SHALL use the catalog default; explicit empty or unknown releases SHALL fail before remote access. Resolution SHALL preserve source configuration and return isolated catalog values. Minimum requirements, desired targets, model applicability and provenance SHALL remain distinct; unknown requirements SHALL retain reasons and never imply qualification. + +#### Scenario: Explicit unsupported release + +- **WHEN** a DPF input selects a release absent from the embedded catalog +- **THEN** configuration fails without substituting the default release or contacting infrastructure. + +#### Scenario: Inventory hardware is outside the BOM applicability + +- **WHEN** identity collection succeeds for hardware outside the selected release BOM +- **THEN** inventory remains useful and a reasoned later-phase compatibility check is unresolved; no supported tuple is invented. + ## Baseline and limits Tag-driven wiring covers root/generate/discover; standalone deploy/validate also use typed Host mappings. Nonzero inference from `Options` cannot represent every explicit zero; use the existing presence-aware inputs. This contract does not promise purity for discovery. [Artifact handoff](../artifact-handoff/spec.md) owns persisted metadata and source selection. diff --git a/openspec/specs/discovery-inventory/spec.md b/openspec/specs/discovery-inventory/spec.md index 4f55f052..a7d839d0 100644 --- a/openspec/specs/discovery-inventory/spec.md +++ b/openspec/specs/discovery-inventory/spec.md @@ -157,6 +157,109 @@ An inventory extension MUST cover fresh discovery and existing-config refresh. R - **WHEN** an existing user config contains custom keys and explicit false values - **THEN** refresh changes the intended inventory/override fields and preserves unrelated user content. +### Requirement: Reasoned DPF observations + +New and changed discovery checks SHALL retain a stable reason code and safe human-readable explanation for failed, unavailable, unknown, skipped and deferred outcomes. Where known, the observation SHALL identify its subject, vantage, tool or resource, and provenance. Missing tools, denied access, malformed output, timeout, absent properties, unsupported capabilities and conflicts MUST NOT collapse into an unexplained not-measured result. Optional failures SHALL remain separate from required physical identity and association failures. + +#### Scenario: Typed DPF device results + +- **WHEN** DPF inventory is serialized as `v1alpha3` +- **THEN** both host and DPU records publish available facts through dedicated domain fields and failures, mismatches and unavailable facts through `issues` entries with `field`, `code`, safe `message` and `tool` where known. Generic observation maps, successful per-value state, raw command strings, vendor firmware dumps and source-reference bookkeeping are omitted. +- **AND** verified PSID, BMC administrator authorization, firmware enumeration completeness, runtime tool versions, per-card firmware/PSID, next-boot configuration and host privileges use named typed fields. Firmware versions retain known running/staged activation, and ERoT firmware remains distinct from BMC firmware. +- **AND** a public device `id` requires successful final identity rechecks; failed candidate serial/model context does not become an eligible identity. Absence of a fact or diagnostic does not establish authorization or readiness. +- **AND** version-aware decoding rejects contradictory facts and preserves identity, authorization, software-check and refresh semantics. Legacy `v1alpha1` and `v1alpha2` remain readable conservatively without promoting absent status to verified identity. +- **AND** host BMC collection retains identity, PCI mappings and advertised reset types without general firmware/time inventory or DPU-only component placeholders. +- **AND** the release BOM retains operator/component artifact coordinates, exact hardware applicability, BFB version/URL/known digest and named documented minimum fields. Omitted minima remain unknown; a BFB target never becomes a measured OS/BSP version. Catalog source/revision lists, generic requirement objects and static limitation prose are not exported or silently substituted from newer catalog defaults. +- **AND** saved `scopeDigest` continues to guard resolved endpoint/trust selection for refresh without exporting a duplicate `requestedScope` list; removal of private provenance alone is not a material inventory change. + +#### Scenario: Firmware tool is absent + +- **WHEN** an SSH endpoint lacks an executable needed for an optional firmware observation +- **THEN** the saved inventory identifies the missing tool and reason, without installing it or discarding successfully collected identity. + +#### Scenario: Optional SSH command times out + +- **WHEN** an optional SSH command times out while the overall discovery context is still active +- **THEN** the command retains a specific unavailable reason and is not retried; subsequent reads may reconnect only to the original explicit endpoint with the same credentials and host-key trust snapshot. +- **AND** reconnection authentication/trust failures, failed required identity rereads, changed identity and overall cancellation still prevent a complete observation. A command timeout never promotes earlier identity into a completed result. + +#### Scenario: Equivalent firmware spellings + +- **WHEN** multiple sources report versions differing only in surrounding whitespace +- **THEN** collection and serialized inventory use the same conflict comparison, retain the measured spellings and activation states, and do not fail publication solely due to that whitespace. +- **AND** a padded measured version remains ineligible to establish a software minimum; different versions without a staged-image distinction retain their conflict. + +#### Scenario: Restricted device rejects a query + +- **WHEN** an installed tool cannot read a restricted device +- **THEN** the collector retains an access/query failure rather than claiming the tool is missing or the hardware absent, without changing privilege settings. + +#### Scenario: DPF serial match contradicts host PCI vendor + +- **WHEN** a same-serial host adapter exposes a non-NVIDIA PCI vendor for a discovered BlueField DPU +- **THEN** the required association fails with a specific vendor-conflict reason and retains both identity sources and PCI evidence; discovery remains partial and preserves the prior complete handoff. +- **AND** a contradictory candidate cannot be discarded to accept another same-serial match. Missing vendor information remains unknown, and unrelated mixed-vendor adapters remain in host inventory. + +#### Scenario: Supported BF4 Redfish PSID sources + +- **WHEN** the supported BF4 reader establishes physical identity and reads System/chassis AssetTags +- **THEN** it persists a typed `psid` Redfish fact on success, or an `issues` entry for `field: psid` on failure, preferring System and falling back to the identity-selected chassis, with authenticated reads and no fabricated SSH/runtime data. +- **AND** absent/null/empty/N/A stays unknown; malformed or contradictory values remain unavailable, and any source-field change or failed required recheck prevents an observed value. Optional PSID gaps do not erase physical identity or independently fail discovery. +- **AND** the PSID fact and eligibility affect inventory fingerprint/diff; consumers requiring PSID must reject invalid/conflicting selected-source evidence and require one observed identifier. + +### Requirement: DPF locator candidates are not identity + +Address lookup SHALL operate only on explicit MAC selectors and bounded declared sources. It SHALL preserve expired leases, ambiguous matches and unavailable sources as reasoned outcomes. Discovered addresses MUST NOT implicitly receive credentials, become selected endpoints or establish physical identity. + +#### Scenario: Sources disagree + +- **WHEN** usable sources associate one requested MAC with different addresses +- **THEN** all candidates retain source provenance and the result is ambiguous; discovery does not choose or connect to one. + +#### Scenario: Lease source cannot be parsed + +- **WHEN** a configured lease file is absent, unreadable, oversized or malformed +- **THEN** its unavailable result retains a safe reason and it contributes no candidates; other source evidence remains separately visible. + +#### Scenario: Host/DPU facts without evidence-reference bookkeeping + +- **WHEN** discovery collects host or DPU inventory through Redfish or SSH +- **THEN** published device records retain named facts and field-specific issues without device evidence records/references or raw vendor source metadata, including complete and partial artifacts. +- **AND** removal of metadata does not remove any authentication, administrator, identity, association, firmware activation, PSID reread, cleanup or publication check. Report checks retain their result states and reasons. +- **AND** legacy inventories containing device source metadata remain readable; refreshed facts replace that metadata without changing material fingerprints solely because the metadata is absent. Original source backups remain unchanged. +- **AND** the discovered YAML omits every evidence map and evidence reference, including nested management, SSH, address-lookup and probe records, while preserving typed facts, failures and legacy readability. The separate JSON diagnostic report and internal collectors retain their source contracts. +- **AND** omission applies to typed diagnostic fields, not arbitrary data-map keys; interface names, DNS domains and user selectors named `evidence`, `evidenceRef` or `evidenceRefs` retain their values. + +### Requirement: DPU BMC administrator evidence + +DPF discovery SHALL require read-only evidence that each discovered DPU BMC's +authenticated session has the full standard Administrator privilege set, with a +uniquely matching local account and verified account state. +It SHALL bind the session username to its advertised own account and linked role, +or identify one matching account from a complete bounded collection, +verify enabled/unlocked/no-password-change state, and reject missing, denied, +malformed, unsafe or inconsistent evidence. Custom roles with the complete standard +privilege set SHALL be accepted; username, role name and OEM privilege names alone +MUST NOT substitute. Host-only and ARM-only observations SHALL remain partial. + +#### Scenario: External authentication collides with a local account + +- **WHEN** a session username matches a local administrator but effective-role or local-authentication evidence is absent or contradictory +- **THEN** discovery fails the administrator check as unverifiable, retains physical observations and preserves the prior complete handoff. +- **AND** matching session Roles or an explicit trusted local session UserAuthenticationSource is required; username equality, current LocalAccountAuth Enabled policy and absent external-provider settings do not prove an existing session's origin. Disabled local authentication and explicit external session sources remain unsupported; Fallback/LocalFirst require an explicit trusted local session source. Malformed exposed fields cannot use an absence fallback. + +#### Scenario: Administrator authorization cannot be established + +- **WHEN** identity reads succeed but account or effective-role evidence is missing, denied, nonadministrator, restricted or requires a password change +- **THEN** discovery retains physical identity evidence with a specific safe administrator failure reason and fails its required discovery check without publishing a complete handoff. +- **AND** discovery neither changes passwords/roles nor tests authorization by mutation; only its normal temporary session creation and cleanup are permitted writes. + +#### Scenario: BF4 B4240 firmware exposes different identity graphs + +- **WHEN** the service advertises NVIDIA and the exact BF4/B4240 product +- **THEN** discovery accepts an NVIDIA B4240 physical Card with a matching System serial, or a serial-bearing B4240 BMC Component reciprocally contained by a physical Card and linked by that System. +- **AND** missing System properties remain absent; conflicting properties, unsafe links, multiple matching candidates or a changed identity graph prevent complete inventory. Free-text descriptions and fuzzy model names do not supply identity. + ## Baseline and limits There is no universal probe registry or typed provenance/unknown framework. Existing trust queries intentionally fall back to unknown. Discovery mutates its supplied config and cluster labels; setup/teardown belongs to [managed effects](../managed-effects/spec.md). Rendering consumes these observations under [profile planning](../profile-planning/spec.md). diff --git a/openspec/specs/outcomes-evidence/spec.md b/openspec/specs/outcomes-evidence/spec.md index 8be86198..7d322eb2 100644 --- a/openspec/specs/outcomes-evidence/spec.md +++ b/openspec/specs/outcomes-evidence/spec.md @@ -34,6 +34,13 @@ A result or gate extension MUST define command execution success separately from - **WHEN** a resource classifier returns in-progress - **THEN** aggregation applies the documented mode policy and does not describe that resource as ready. +#### Scenario: DPF inventory without management readiness + +- **WHEN** explicit DPF inventory sources establish physical identities and unique host/card associations, each DPU BMC account has verified administrator privileges, and required observation, cleanup and publication succeed +- **THEN** inventory discovery can complete without a management Kubernetes cluster; release qualification and provisioning authentication remain separate, visible later-phase checks. +- **AND** unavailable selected sources and ambiguous identities still fail inventory; an observed BMC address never authorizes sending credentials to it automatically. +- **AND** host-only or ARM-only input preserves partial identity evidence and exits nonzero because DPU BMC administrator authorization is missing; it cannot publish a completed handoff. + ### Requirement: Output compatibility Output extensions SHALL preserve command-specific framing, existing public field meanings, diagnostic separation and exit mapping unless a compatibility change is explicit. Text, JSON and HTML MUST describe compatible evidence without assuming identical envelopes. @@ -48,6 +55,14 @@ Output extensions SHALL preserve command-specific framing, existing public field - **WHEN** a new family is emitted - **THEN** existing result/report consumers retain existing families and can identify the added family without redefining older field meanings. +#### Scenario: DPF operator-facing progress and results + +- **WHEN** DPF discovery runs with normal CLI output +- **THEN** live operation/check status is emitted as work occurs on stderr, identifies the selected source by name/address, distinguishes running/passed/failed/unavailable states, and does not announce successful identity or cleanup before its required checks complete. +- **AND** the final text summary distinguishes discovery blockers, later provisioning prerequisites and checks not requested; internal identity/comparison hashes remain in structured artifacts rather than replacing source names. Host/DPU records from either Redfish or SSH omit device evidence maps/references and publish named facts with field-specific `issues`; unsuccessful reads do not require a generic observation wrapper. +- **AND** JSON remains one final stdout document, quiet suppresses progress, and typed `v1alpha3` inventory presentation preserves required report check states, usable facts, failure reasons, acceptance and process exit status. Legacy `v1alpha1`/`v1alpha2` decoding cannot infer missing verification; omitted BOM minima remain unresolved, and private provenance removal alone cannot create a material diff. A final output-write failure remains a command failure. +- **AND** debug/trace uses the existing logging controls and bounded operation metadata; credentials, session tokens, raw vendor responses, SSH command output and secret-file contents are not logged. + ### Requirement: Evidence existence and partial results An advertised report path SHALL identify an artifact that was successfully written. Extensions MUST safely handle partial initialization and retain useful failure evidence; unavailable reports SHALL be reported as unavailable rather than advertised as complete. diff --git a/openspec/specs/platform-support/spec.md b/openspec/specs/platform-support/spec.md index 05540204..62ce1273 100644 --- a/openspec/specs/platform-support/spec.md +++ b/openspec/specs/platform-support/spec.md @@ -75,4 +75,5 @@ Support claims MUST identify the platform, profile, hardware and phases actually - Implementation: [flavor](../../../pkg/config/flavor.go), [targets](../../../pkg/target), [profiles](../../../pkg/profiles), [OCP configuration](../../../pkg/networkoperatorplugin/ocp_config.go). - Existing suites: [Host driver](../../../pkg/target/host/driver_test.go), [OCP config](../../../pkg/networkoperatorplugin/ocp_config_test.go), [OCP validation](../../../pkg/networkoperatorplugin/ocp_validate_test.go). -- User documentation: [OpenShift prerequisites and qualification](../../../docs/user/openshift.md), [targets](../../../docs/advanced/targets.md). +- User documentation: [OpenShift prerequisites and qualification](../../../docs/user/openshift.md). +- Engineering: [target extension](../../../engineering/advanced/targets.md). diff --git a/openspec/specs/workflow-execution/spec.md b/openspec/specs/workflow-execution/spec.md index b08c147f..2fb9a227 100644 --- a/openspec/specs/workflow-execution/spec.md +++ b/openspec/specs/workflow-execution/spec.md @@ -62,6 +62,20 @@ Equivalent entry points SHALL reuse common semantics for equivalent modes. Inten - **WHEN** standalone deploy runs with dry-run selected - **THEN** its Kubernetes deployment path uses server-side dry-run semantics; this does not describe earlier discovery effects. +### Requirement: Actionable non-success outcomes + +New and changed workflow checks SHALL persist a stable reason code and safe explanation for every failed, unavailable, unknown, skipped or deferred result. The result SHALL distinguish missing prerequisites from an attempted operation failure and identify the subject and source where known. Diagnostic persistence MUST NOT disclose credentials or treat optional unknowns as passes. Existing unrelated baseline paths are not retroactively migrated by this requirement. + +#### Scenario: Prerequisite executable is missing + +- **WHEN** a workflow cannot execute a check because a required tool is unavailable +- **THEN** the report and saved observations identify the executable and explain the missing prerequisite, while required versus optional policy determines whether dependent work may continue. + +#### Scenario: Service does not exist in the current phase + +- **WHEN** a check depends on an operator-created endpoint before that operator is installed +- **THEN** it is deferred with a reason and required phase, not reported as passed or used to fail initial hardware inventory. + ## Baseline and limits The root pipeline runs discover/generate/deploy, not automatic validation. Exactly-once dispatch is not a promise of global idempotency or a non-reusable operation object. Discovery may mutate supplied config and use a global logger. It performs pre-clean before some worker validation and defers cleanup only after successful setup. See [effects](../managed-effects/spec.md), [artifacts](../artifact-handoff/spec.md) and [outcomes](../outcomes-evidence/spec.md) for those independent responsibilities. @@ -70,4 +84,4 @@ The root pipeline runs discover/generate/deploy, not automatic validation. Exact - Implementation: [target registry](../../../pkg/target), [Host adapters](../../../pkg/target/host), [CLI routing](../../../pkg/cmd/target_cli.go), [application deploy](../../../pkg/app/deploy.go). - Existing suites: [target](../../../pkg/target/target_test.go), [Host driver](../../../pkg/target/host/driver_test.go), [deploy](../../../pkg/target/host/deploy_test.go), [launcher](../../../pkg/target/host/launcher_test.go). -- User documentation: [targets](../../../docs/advanced/targets.md). +- Engineering: [target extension](../../../engineering/advanced/targets.md). diff --git a/openspec/workflows/README.md b/openspec/workflows/README.md new file mode 100644 index 00000000..a7ef37e1 --- /dev/null +++ b/openspec/workflows/README.md @@ -0,0 +1,64 @@ + + +# Procedural documentation sources + +`dpf-discovery.json` owns the user steps and flow transitions for initial DPF +inventory, address-candidate lookup and refresh. Its generated page is +`docs/user/dpf-workflows.md`; edit the JSON, not the generated Markdown. + +This is a **repository-owned documentation extension**, not a native executable +OpenSpec workflow format. It does not add a runtime state machine or a discovery +plugin. Python's standard library is used only during documentation generation +and tests; the Go CLI and remote collection do not depend on Python. Commands +inside step bodies are displayed as examples and are never executed by this tool. + +## Regenerate and verify + +From the repository root: + +```sh +python3 scripts/generate-workflow-docs.py +python3 -m unittest discover -s scripts -p 'test_generate_workflow_docs.py' -v +python3 scripts/generate-workflow-docs.py --check +``` + +The Docs CI job runs the tests and `--check` before strict MkDocs. The existing +`check-docs.py` CLI example checks still run separately in code CI. The generator +is deterministic, omits timestamps, and rejects invalid metadata before writing. +`--check` reports missing/stale output with exit 1 and leaves output untouched. +`--source` and `--output` can select explicit paths for local testing. + +## Format version 1 + +The source is one JSON object with `formatVersion: 1`, `title`, `introduction`, +and a nonempty `workflows` array. An optional `$comment` carries source licensing. +Each workflow has a unique `id`, `title`, `introduction`, and nonempty `steps`. +Each step has an `id` unique within that workflow, `title`, Markdown `body`, and +`transitions`. Each transition has a destination step `to` and a nonempty `label`. +An empty transition list ends the procedure. Retry transitions may point backward. +IDs use lowercase letters, digits and hyphens and start with a letter. The first +listed step is the starting step; list steps in the order users should read them. + +The generator validates field shapes, duplicate workflow/step IDs, duplicate JSON +fields, and unknown or duplicate transition destinations. It renders the same +transitions as both Mermaid edges and next-step links. The source remains a human +procedure: no conditions are evaluated, no infrastructure is inspected, and graph +structure does not prove the commands implement the contract. Review the source, +rendered prose, examples and diagram together against the actual CLI behavior. + +## Maintaining a workflow + +1. Update applicable [OpenSpec contracts](../README.md) when the shared promise + changes, and implement/test the CLI behavior. +2. Edit the procedure's inputs, actions, failure explanations and transitions in + the JSON. Keep required site inputs separate from defaults and observations. +3. Regenerate and run generator tests, drift detection, CLI example checks and + strict MkDocs. Include the source and generated page in the same change. +4. Review runtime/hardware evidence separately. Inventory success, deployment + readiness and hardware qualification are distinct outcomes. + +The generator currently targets this one DPF procedure document. Extending it to +other guides is a future scoped documentation change, not a workflow execution API. diff --git a/openspec/workflows/dpf-discovery.json b/openspec/workflows/dpf-discovery.json new file mode 100644 index 00000000..8f5ad9c1 --- /dev/null +++ b/openspec/workflows/dpf-discovery.json @@ -0,0 +1,249 @@ +{ + "$comment": "Copyright 2026 NVIDIA CORPORATION & AFFILIATES; SPDX-License-Identifier: Apache-2.0", + "formatVersion": 1, + "title": "DPF discovery workflows", + "introduction": "Use these procedures with the experimental `l8k discover --target dpf` CLI. The DPU estate is separate from Host Network Operator deployment. Start with **inventory mode**; management probe mode is unavailable in this preview. Discovery collects facts and writes a handoff. It does not create VMs, install Kubernetes or DPF, configure DPUServices, change firmware, enroll trust, or start a DPF deployment.\n\nFor complete field and error details, see [DPF discovery reference](dpf-discovery.md). Each diagram follows the same steps as its procedure. Commands are examples to run deliberately from a machine with direct access to your selected endpoints.", + "workflows": [ + { + "id": "initial-inventory", + "title": "Collect the first inventory", + "introduction": "Use this workflow when you know the host access addresses. Explicit DPU BMC endpoints and verified administrator privileges are required for a complete inventory. Without explicit DPU endpoints, Launch Kit derives BlueField card records from host inventory and leaves their access bindings unknown; host-only or ARM-only discovery retains partial evidence and exits nonzero.", + "steps": [ + { + "id": "choose-inputs", + "title": "CLI machine: choose sources and access inputs", + "body": "Choose each source explicitly: a host operating system over SSH or its BMC over Redfish; choose the DPU BMC over Redfish for a complete run. DPU ARM SSH is available for partial evidence but cannot verify BMC administrator privileges. A selected source must succeed; Launch Kit does not silently replace failed Redfish access with SSH. Host SSH can supply host-to-card identity in Zero Trust when PCI/VPD access is available. ARM SSH needs a booted, reachable DPU operating system. Neither access path proves trusted DPU state.\n\n| Input | What you supply or allow Launch Kit to determine |\n| --- | --- |\n| Always required | At least one host endpoint and explicit DPU BMC endpoints with administrator credentials for completion; the source type; a credential alias, username, and password reference or SSH key reference for each selected endpoint. Addresses and credentials are not guessed. |\n| Required for SSH | A verified `knownHostsFile`; direct connectivity from the CLI machine. SSH port defaults to `22`; ambient SSH config, agents and ProxyJump are not used. |\n| Conditional | DPU ARM SSH for partial OS measurements; a private CA for Redfish when system trust is insufficient; `systemURI` if a BMC exposes several Systems. |\n| Optional | A management kubeconfig for additional observations. Inventory needs none, but an explicitly supplied kubeconfig must work. |\n| Defaultable | `dpf.release` defaults to `26.4.0`; Redfish certificate verification is enabled; SSH `sudo` defaults to `false`. Choose `discovery.probes.mode: inventory` explicitly because the default management mode is unavailable in this preview. |\n| Discovered | Host/system and card identities, host-to-DPU associations, visible hardware/software/network observations, and optional local BMC address hints. |\n\nUse an account whose own session proves its effective administrator role. If session Roles is absent, local authentication must be established by an explicit trusted local session `UserAuthenticationSource`. A current `LocalAccountAuth: Enabled` policy alone cannot prove how an existing session authenticated. Disabled local authentication and explicitly external session sources are unsupported; Fallback/LocalFirst require the explicit local session source. Missing origin plus missing Roles cannot inherit privileges from a same-named local administrator. Discovery reports unverifiable proof as partial without replacing a complete handoff.\n\nDo not provide product versions as measured facts. The selected release supplies the software bill of materials (BOM); discovery measures installed versions separately. Omitted or null `dpf.release` uses the pinned default; an empty or unknown release is rejected.\n\nAll network connections start from the machine where you run `l8k`. Host and DPU endpoints do not need to connect to each other for this workflow.\n\n| Connection from the CLI machine | Where the operation runs | What it provides |\n| --- | --- | --- |\n| SSH to the selected host | Host operating system | Host identity, installed card serials and available host/card facts |\n| SSH to the selected DPU ARM address | DPU operating system | DPU identity, local hardware/network facts, and optionally its BMC address through local IPMI |\n| HTTPS to a selected host or DPU BMC | That BMC's Redfish service | System/card identity, host PCI/reset information and available named DPU firmware versions |\n| HTTPS to the management API, only if a kubeconfig is supplied | Existing management Kubernetes API | Cluster identity, version and nodes, plus optional storage and advertised DPF API information |\n| No remote connection | CLI machine | Read input/trust/lease files, match identities, compare software requirements and save results |\n\nChoose SSH or Redfish separately for each endpoint. Selecting host Redfish does not run tools in the host OS. Selecting DPU SSH does not log in to DPU Redfish. The [collection step](#initial-inventory-collect) lists the operations for each choice.", + "transitions": [ + { + "to": "prepare", + "label": "Sources selected" + } + ] + }, + { + "id": "prepare", + "title": "CLI and endpoint administrators: prepare access", + "body": "Put passwords in an environment variable or a private file, and SSH private keys and known-hosts entries in files under your control. Reference them from YAML; do not put password values in command-line arguments. File references are relative to that YAML file. Verify SSH keys and Redfish CA trust through your normal provisioning process.\n\nFor SSH inventory, ask the endpoint administrator to make the required read tools and permissions available before the run. Host identity/PCI collection uses tools such as `dmidecode` and `lspci`; DPU identity additionally uses `ipmitool`. Optional hardware checks use `mlxfwmanager`, `mlxconfig`, `mlxprivhost` and `systemctl`; network checks use existing `ip`, resolver and LLDP tools where available. Tool availability and access differ between host and ARM operating systems.\n\nSet `ssh.sudo: true` only where noninteractive elevated reads are authorized and configured. Launch Kit then uses `sudo -n`; it never prompts for a sudo password, installs tools, changes privileges or repairs device settings. Missing tools, denied permissions and unsupported or malformed outputs keep specific reason codes and explanations. Required identity failures stop a complete inventory; optional unknown observations remain visible for review.\n\nPrepare a DPU BMC account that is enabled, unlocked and does not require a password change, with all five standard Administrator privileges. A custom role with those privileges is accepted; names alone are insufficient. Discovery verifies the session, unique matching account and linked role using read requests. Handle password changes separately before rerunning; discovery never rotates passwords or tests privileges through writes.\n\nPasswords may use an environment reference supplied for the invocation or a `passwordRef.file` pointing to an owner-only UTF-8 password file. Environment variables are convenient for automation but can be inherited or exposed by diagnostics. File references avoid that export but remain plaintext files: keep them private and outside version control. Discovery saves references, never secret values.", + "transitions": [ + { + "to": "configure", + "label": "Access prepared" + } + ] + }, + { + "id": "configure", + "title": "CLI machine: create the inventory configuration", + "body": "Save this example as `dpf-site.yaml`, replacing the documentation address and credential paths with your host values. It uses host SSH and DPU BMC Redfish and requires no Kubernetes cluster. Ensure the listed files exist before running.\n\n```yaml\ndpf:\n release: 26.4.0\ncredentials:\n host-reader:\n username: inventory\n privateKeyRef: {file: ./credentials/host-key}\n dpu-maintainer:\n username: bmc-maintainer\n passwordRef: {env: DPU_BMC_PASSWORD}\ndiscovery:\n hosts:\n - credentialRef: host-reader\n ssh:\n address: 192.0.2.21:22\n knownHostsFile: ./credentials/known_hosts\n sudo: true\n dpus:\n - bmcEndpoint: https://192.0.2.11\n credentialRef: dpu-maintainer\n probes:\n mode: inventory\n```\n\nFor DPU ARM observations, use the SSH shape instead; ARM-only DPU input produces partial inventory with a nonzero exit because it cannot prove BMC administrator privileges. For Redfish, use `bmcEndpoint` and a password credential reference instead of `ssh`; see the [complete access examples](dpf-discovery.md#inputs-and-invocation). Each endpoint has exactly one access source. Plain YAML is the CLI input, not a Kubernetes object. Do not copy previously observed `inventory` facts into a new seed as user assertions.", + "transitions": [ + { + "to": "run", + "label": "Configuration ready" + } + ] + }, + { + "id": "run", + "title": "CLI machine: start discovery", + "body": "Choose a new output path for the first run. Run from a machine that can reach every selected endpoint and read the referenced files.\n\n```bash\nl8k discover --target dpf --user-config dpf-site.yaml \\\n --save-cluster-config dpf-discovered.yaml --output json \\\n > discovery.json 2> discovery.log\n```\n\nRecord the process exit status as well as the files. `discovery.json` contains the final result; progress and diagnostics go to `discovery.log`. The command creates its own run directory next to the requested YAML. It does not send credentials to addresses found during collection. Redfish uses a temporary login session and deletes that session; SSH runs read commands on explicitly selected endpoints.\n\nWhile this command runs, Launch Kit performs the collection and matching steps below automatically. You do not need to run the listed remote commands yourself. The example contacts the host over SSH and the explicitly selected DPU BMC over Redfish. Host-only input contacts no DPU endpoint and returns partial evidence with a nonzero exit.\n\nWatch live source/check status on stderr: RUNNING is emitted before the operation, followed by its result. Endpoint names/addresses distinguish concurrent work. To keep progress visible with JSON, redirect stdout only (`>discover.json`). Add `--log-level debug` for stage timing or `--log-level trace` for request/tool metadata; `2>discover.log` saves these messages instead of displaying them. Quiet suppresses progress.", + "transitions": [ + { + "to": "collect", + "label": "Collect from selected endpoints" + } + ] + }, + { + "id": "collect", + "title": "Selected host, DPU and BMC: read identity and facts", + "body": "Launch Kit uses the configured connection for each endpoint. Required identity reads and DPU BMC administrator verification must succeed to complete inventory. Optional reads add facts; a missing tool, denied operation or unsupported response leaves an explained gap instead of silently passing the check. After collecting optional facts, Launch Kit rereads identity to detect a changed or replaced device. If an optional SSH command times out, Launch Kit retains the reason and reconnects to the same configured endpoint with the same credentials and verified host keys before the next read. It does not retry the timed-out command. Failed reconnection, a failed identity reread or an expired overall deadline still prevents complete inventory.\n\n**On a host selected through SSH**\n\n| Read performed on the host | Purpose and result | Required or optional |\n| --- | --- | --- |\n| `dmidecode -s system-uuid` | Identifies the physical host. | Required |\n| `lspci -Dnn -d 15b3:` and per-card `lspci -D -vv -s ` | Finds NVIDIA network cards, their PCI locations and available card serial/slot information. | PCI enumeration is required; a selected DPU needs a usable serial for host-to-card matching. Missing serials remain visible. |\n| Linux PCI device files | Reads each function's network interface, driver and NUMA location. | Optional |\n| `systemctl show rshim --property=LoadState --property=ActiveState` | Reports whether the host's RShim service is present and active. It does not start the service or use it to access a DPU. | Optional |\n\n**On a DPU ARM operating system selected through SSH**\n\n| Read performed on the DPU ARM OS | Purpose and result | Required or optional |\n| --- | --- | --- |\n| `dmidecode -s system-serial-number` and `ipmitool fru print 0` | Reads system and built-in card records; their serials must agree. Also records available model/board information. | Required for DPU identity |\n| `ipmitool lan print` | Reads the local BMC's configured LAN address over local IPMI. | Optional |\n| `lspci -Dnn -d 15b3:` and per-card `lspci -D -vv -s ` | Locates ARM-visible NVIDIA cards and available card serials before associating hardware readings with them. | Optional |\n| `mlxprivhost -d q --full` | Reports host privilege restrictions, such as restricted access or disabled RShim, for a visible BlueField. | Optional |\n\nLocal IPMI is an operation **from the DPU ARM OS to its local BMC**, not an HTTPS login from the CLI. The returned BMC address is recorded, but Launch Kit does not connect to it or send it credentials. Host and ARM PCI addresses are local to different operating systems and need not match.\n\n**On each selected host or DPU ARM OS, through SSH**\n\n| Read performed on that OS | Purpose and result |\n| --- | --- |\n| `mlxfwmanager --version` | Records the installed MFT tool version. |\n| `mlxfwmanager --query -d ` | Queries the specific visible card's current NIC firmware and PSID. |\n| `mlxconfig -d q ` | Reads `INTERNAL_CPU_OFFLOAD_ENGINE`, `INTERNAL_CPU_RSHIM` and `NUM_OF_VFS` on visible BlueFields. These results are next-boot settings, not proof of the running mode. |\n| `hostname`, `cat /etc/os-release`, `ip -j address show`, `ip -j -4 route show` | Reports the OS, local addresses and IPv4 routes. |\n| `ip -j link show`, `ip -j -d link show`, `ip -j neighbor show` | Reports MACs, bridge/member relationships, available link details and neighbors already known to that OS. It does not scan the subnet. |\n| `cat /etc/resolv.conf`, `readlink -f /etc/resolv.conf`, `resolvectl dns`, `resolvectl domain` | Reports configured DNS servers, search domains, per-link DNS and the resolver configuration in use. |\n| Selected `server=` and `no-resolv` settings in `/etc/dnsmasq.conf` and `/etc/dnsmasq.d/*.conf` | Reports local dnsmasq DNS forwarding configuration. This is separate from reading a DHCP lease snapshot on the CLI machine. |\n| `lldpctl -f keyvalue` | Reports neighbors already available from the local LLDP service. |\n\nThese hardware and network reads are optional. `` comes from that same OS's card inventory; Launch Kit queries each card separately. In Zero Trust, the host may be unable to query a restricted card even when `mlxfwmanager` is installed. That limitation is reported; it does not cause an automatic connection to the DPU. Tools must be installed on the OS where the command runs, not merely on the CLI machine. When configured, protected hardware reads use `sudo -n`; tools are never installed or settings changed by discovery.\n\n**On a selected host or DPU BMC, through HTTPS Redfish**\n\nThe CLI creates a temporary session and reads system and card inventory. Host BMC reads locate cards associated with the selected host and retain advertised reset types; they do not collect host firmware/time/NTP or DPU-only component placeholders. DPU BMC reads identify that DPU, query available firmware components and verify the authenticated session account and linked role through AccountService. All five standard Administrator privileges and enabled/unlocked/no-password-change state are required; denied or unverifiable evidence fails closed. The supported BF4/B4240 reader also records an optional PSID from the selected System or identity-selected chassis AssetTag, with authenticated reads and a final reread. Missing PSID stays unknown; invalid, contradictory or changed fields remain unavailable for consumers requiring PSID. No PSID is guessed from the model. No SSH commands run in either OS for this choice. System/card identity is required; optional component versions may remain unknown. A firmware version is not assumed to be running unless the BMC explicitly identifies it as active. The CLI deletes its session afterward and reports any cleanup failure. It does not change passwords, roles, firmware, NTP, power or trust settings.\n\n**At the management Kubernetes API, if configured**\n\nThe CLI reads the cluster identity from `kube-system`, the Kubernetes version and the node list, including control-plane membership and reported readiness. These reads are required when you supply a management kubeconfig. StorageClasses and advertised DPF API groups are additional optional reads. Inventory mode creates no Pods and performs no management-network connectivity tests. It does not install DPF or run commands inside Kubernetes nodes.", + "transitions": [ + { + "to": "match", + "label": "Return facts and read failures to CLI" + } + ] + }, + { + "id": "match", + "title": "CLI machine: match cards and assess results", + "body": "Launch Kit compares physical card serials returned by the selected sources. Each explicit DPU must match exactly one adapter on one observed host. A missing serial, conflicting identity or multiple matches prevents a complete inventory; an IP address or PCI address cannot substitute for that match. With host-only input, the inventory lists host-reported BlueFields without claiming a working ARM or BMC connection. These records remain partial and fail the required DPU BMC administrator check; ARM SSH cannot bypass that requirement.\n\nNext, the CLI compares eligible measured versions with the selected release's documented requirements. It reports unavailable measurements and unknown requirements separately from below-minimum versions. These comparisons do not install or update software.\n\nIf address lookup is configured, the CLI separately matches your supplied management MACs to local lease-file records and neighbors read from the selected SSH endpoints. These are unverified IP candidates, not card identities or host-to-DPU associations. No candidate is contacted automatically.\n\nFinally, the CLI checks collection and session-cleanup results, compares against any previous inventory, and writes the YAML and report. Required failures produce partial output; optional gaps remain recorded even when inventory completes. All output files are written on the CLI machine.", + "transitions": [ + { + "to": "inspect", + "label": "Review saved results" + } + ] + }, + { + "id": "inspect", + "title": "CLI machine: review inventory and reasons", + "body": "The normal text summary shows the required-check result and named host/DPU mappings. It separates discovery blockers from later provisioning prerequisites and checks not requested. Repeated qualification/BOM gaps are grouped by affected sources; they do not mean a hardware failure was detected. Internal identity/comparison SHA identifiers remain in JSON/YAML, not as operator-facing device names. Both host and DPU results use named facts and field-specific issues; device evidence references, raw command strings and vendor firmware metadata are omitted. Saved YAML omits evidence maps and references from every source, including management, SSH, address lookup and connectivity probes. Diagnostic JSON is separate and can retain internal bookkeeping.\n\nInspect `discovery.outcome`, `discovery.checks`, `errors`, `artifacts` and `acceptance` in the JSON report. Review saved `inventory.hosts`, `inventory.dpus`, `inventory.associations` and field-specific issues before using the handoff. Complete discovery writes `dpf-discovered.yaml`; partial discovery preserves an existing handoff and writes `partial.yaml` and `report.json` under `dpf-discovered.discovery//`. Use the report's artifact paths instead of guessing the run ID.\n\nRead each host/DPU `issues` entry's `field`, `code`, `message` and `tool` where present. Successful values live in named fields such as `psid`, `bmcAdministratorVerified`, `runtime.mftVersion` and `runtime.hardware.cards[].firmwareVersion`. Cards also retain typed `nextBoot` settings and `hostPrivileges`; next-boot values do not prove the running mode. A public `id` requires successful final identity rechecks; a serial retained on a failed record is diagnostic context. Generic observation maps and repeated successful state are omitted.\n\nDPU `firmware` groups version lists by component, retaining known activation separately. `firmwareInventoryComplete: true` means enumeration completed, not that every version is known or compatible. ERoT firmware remains separate from BMC firmware. The JSON report retains all required check outcomes and deferred provisioning checks. New inventories use `v1alpha3`; the updated CLI also reads legacy `v1alpha1`/`v1alpha2` without inferring missing verification.\n\nA missing `mlxfwmanager` is an unavailable prerequisite, not proof of incompatible NIC firmware. A denied Redfish firmware read differs from a property that is not exposed. Conflicting versions or unknown activation cannot establish a software minimum.\n\n| Example reason | What to do |\n| --- | --- |\n| `TOOL_NOT_AVAILABLE` / `PERMISSION_DENIED` | Check the named endpoint tool or authorized noninteractive sudo policy; do not treat the unmeasured check as passed. |\n| `ACCESS_DENIED` / `PROPERTY_NOT_EXPOSED` | Distinguish an account permission problem from a property the service does not advertise. |\n| `SOFTWARE_VERSION_CONFLICT` / `SOFTWARE_IDENTITY_UNAVAILABLE` | Reconcile version/activation sources or the rejected physical identity before trusting software comparisons. |\n| `SOFTWARE_BELOW_MINIMUM` | Review the release requirement and plan a separate maintenance action; discovery does not update firmware. |\n| `HOST_ADAPTER_VENDOR_CONFLICT` | Inspect the selected endpoints and matching serial/PCI evidence: a same-serial host adapter reports a non-NVIDIA vendor. Correct the source and rediscover; do not edit observed identity or discard a contradictory candidate to force a match. |\n| `BMC_ADMIN_REQUIRED` / `BMC_ADMIN_ACCESS_DENIED` | Use an authorized account with all standard Administrator privileges and readable account/role evidence. |\n| `BMC_ADMIN_UNVERIFIABLE` / `BMC_ADMIN_NOT_VERIFIED` | Supply explicit trusted DPU BMC access and inspect session Roles, authentication origin and linked account/role evidence. Host/ARM SSH and a same-named local administrator cannot substitute for verified session privileges. |\n| `BMC_PASSWORD_CHANGE_REQUIRED` / `BMC_ADMIN_ACCOUNT_RESTRICTED` | Ask the administrator to resolve password-change, disabled or locked state separately, then rerun. Discovery does not repair access. |\n| `PUBLICATION_FAILED` | Read the error explanation, resolve the local output/path problem, and rerun; only listed successfully written artifacts exist. |\n\nThe resolved software BOM is saved under `inventory.software`: operator/component coordinates, exact hardware models and BFB version/URL/known digest, with documented minima in named fields such as `minimumNICFirmware` and `minimumHostMFT`. Missing minima remain unknown; catalog sources/revisions and generic requirement lists are omitted. A missing artifact digest is not fabricated. Release `26.4.0` has exact BlueField-3 model entries `B3210`, `B3220` and `B3240`; generic family names and BlueField-4 are not sufficient for BOM applicability. The documented NIC/MFT minima can be evaluated when current measurements associated with the correct device exist. BMC, CEC, ATF, UEFI, OS/BSP and golden-image minima remain unknown where the catalog has no supported requirement. A BFB target is a desired artifact, not an observed OS/BSP version. An unknown requirement must remain unresolved; supplying a guessed minimum does not establish compatibility.\n\nExit zero and `discovery.outcome: complete` mean inventory collection, identity/mapping, cleanup and publication succeeded. `acceptance: not-evaluated` remains explicit. Hardware support, DPF authentication, service connectivity and deployment still need separate assessment; DPF generate/deploy/validate are unavailable in this preview.", + "transitions": [ + { + "to": "finish", + "label": "Complete inventory reviewed" + }, + { + "to": "resolve", + "label": "Required failure or conflict" + } + ] + }, + { + "id": "resolve", + "title": "Endpoint owner or CLI operator: correct the cause", + "body": "Use the specific recorded reason to correct the source configuration or ask the infrastructure owner to repair missing tools, permissions, credentials, connectivity or conflicting identity. Do not delete unknown observations or label them passed. Optional gaps may remain documented without blocking inventory; required source failures, identity ambiguity, cleanup or publication errors need resolution. If cleanup is uncertain, use the reported session/resource ownership evidence before any manual action. Rerun after correcting the cause.", + "transitions": [ + { + "to": "prepare", + "label": "Retry after correction" + } + ] + }, + { + "id": "finish", + "title": "CLI machine: retain the inventory and report", + "body": "Retain the seed, discovered YAML, JSON report and stderr log together. The discovered YAML preserves configuration intent and replaces observed inventory; it is not a deployment-ready profile or a provisioning approval. If a DPU access address is missing, use [address-candidate lookup](#address-candidates). For later remeasurement of the same estate, use [refresh](#refresh-inventory).", + "transitions": [] + } + ] + }, + { + "id": "address-candidates", + "title": "Find and verify address candidates", + "introduction": "Use this optional inventory addition when you know an ARM or BMC management MAC but need its possible IP address. It is a locator workflow, not authentication or identity discovery. You still need the host configuration from the first workflow; this is not a standalone subnet scanner.", + "steps": [ + { + "id": "locators", + "title": "CLI machine: prepare MACs and lease snapshots", + "body": "Obtain the exact ARM or BMC management MAC from authoritative asset records or a trustworthy observed interface. You must choose its role (`arm` or `bmc`) and a unique name. Launch Kit does not infer a management MAC from a NIC serial, data-port MAC, hostname or neighboring address.\n\nOptionally obtain a current dnsmasq IPv4 lease-file snapshot through your authorized DHCP administration process and place it on the CLI machine. Launch Kit reads this file locally; it does not log in to a DHCP server or query arbitrary vendor lease APIs. With no lease files, it can use suitable neighbors observed on the SSH endpoints already selected for inventory. Only `REACHABLE` or `PERMANENT` neighbors are candidates; stale entries are excluded.", + "transitions": [ + { + "to": "configure", + "label": "MAC and source selected" + } + ] + }, + { + "id": "configure", + "title": "CLI machine: configure address lookup", + "body": "Merge this fragment into the existing `discovery` section of a seed without saved inventory. Replace the MACs and file path; omit `sources` if you only want eligible observed neighbors. Do not replace the required host list.\n\n```yaml\ndiscovery:\n addressLookup:\n sources:\n - file: ./inputs/dnsmasq.leases\n format: dnsmasq-leases\n devices:\n - name: dpu-a-arm\n role: arm\n mac: '02:00:00:00:00:11'\n - name: dpu-a-bmc\n role: bmc\n mac: '02:00:00:00:00:12'\n```\n\nThe supported file contains the five dnsmasq IPv4 columns: expiry epoch, MAC, IP, hostname and client ID. An expiry of `0` means a nonexpiring record; expired records are not candidates. Each file must be readable, at most 1 MiB and at most 4096 records. A malformed file contributes no candidates and retains a source failure reason. Other DHCP formats are unsupported.", + "transitions": [ + { + "to": "run", + "label": "Selectors configured" + } + ] + }, + { + "id": "run", + "title": "CLI machine: run inventory and local lookup", + "body": "Save the configured seed as `dpf-lookup.yaml` and select a new handoff path.\n\n```bash\nl8k discover --target dpf --user-config dpf-lookup.yaml \\\n --save-cluster-config dpf-candidates.yaml --output json \\\n > candidates.json 2> candidates.log\n```\n\nThe selected endpoints undergo the same [identity and optional reads](#initial-inventory-collect) as first inventory. On the CLI machine, address lookup then reads the configured lease snapshots and matches the supplied MACs against those records and the neighbors returned by remote `ip -j neighbor show`. No candidate is contacted and no credential is sent to it.\n\nWatch live source/check status on stderr: RUNNING is emitted before the operation, followed by its result. Endpoint names/addresses distinguish concurrent work. To keep progress visible with JSON, redirect stdout only (`>discover.json`). Add `--log-level debug` for stage timing or `--log-level trace` for request/tool metadata; `2>discover.log` saves these messages instead of displaying them. Quiet suppresses progress.", + "transitions": [ + { + "to": "review", + "label": "Candidate report available" + } + ] + }, + { + "id": "review", + "title": "CLI operator: review candidate addresses", + "body": "Read `inventory.addressLookup.candidates` for the address, selected name/role/MAC and source. Read `inventory.addressLookup.observations` for each device and lease source. A single address is `ADDRESS_UNVERIFIED`; several addresses for a MAC remain ambiguous, and missing/expired/unreadable evidence keeps its reason. A candidate is not a host-to-DPU association and is never a readiness pass.\n\nUse your DHCP/network or asset owner to resolve multiple, missing or contradictory locators. Do not choose an address merely because it is the first result. Independent identity and endpoint trust are still required even for one candidate.", + "transitions": [ + { + "to": "bind", + "label": "Ownership can be independently verified" + }, + { + "to": "locators", + "label": "Missing or ambiguous evidence" + } + ] + }, + { + "id": "bind", + "title": "Endpoint owner and CLI operator: verify and select", + "body": "Verify endpoint ownership and obtain SSH known-hosts trust or Redfish CA trust and the appropriate account through your normal process. Create a new seed and add the verified endpoint under `discovery.dpus`, with its credential reference and trust policy. Use a new output when changing scope; do not edit the old observed identity to force a match.\n\nRun [initial inventory](#initial-inventory) for that explicit endpoint. Only its authenticated physical-card observation and unique host-card association can establish the binding. A discovered IP alone never authorizes Redfish access or proves Zero Trust attestation.", + "transitions": [] + } + ] + }, + { + "id": "refresh-inventory", + "title": "Refresh an existing inventory", + "introduction": "Use this workflow to remeasure the same selected estate, compare facts and preserve diagnostic history. A refresh is another live read; old successful observations do not stand in for unavailable current measurements.", + "steps": [ + { + "id": "scope", + "title": "CLI machine: check inputs and trust before refresh", + "body": "Start from the existing discovered YAML and keep its credential/trust references valid. Decide whether you are remeasuring the same host/DPU endpoints and trust policy. If endpoint selection, trust policy or physical identity has changed, create a fresh input without old `inventory` and use a new output path. Investigate unexpected replacement or ownership changes before accepting them.\n\nThe saved `scopeDigest` guards resolved endpoint/trust selection; the inventory does not repeat the configured policy list as `requestedScope`. Removing private device provenance alone is not a physical change.\n\nChanging Redfish certificate verification or the contents of a custom CA file requires a new inventory. Moving an unchanged CA file to another path does not. Keep the CLI machine's operating-system trust store maintained; discovery does not report changes within that store.\n\nThe CLI reads legacy `v1alpha1` and `v1alpha2` inventories conservatively, but some older identity/trust bindings cannot support safe refresh. If the command reports that the old inventory is incompatible, verify endpoint ownership, create a fresh input without saved `inventory`, and use a new output path. Keep the previous file as history; do not edit its recorded identities or trust information to bypass the error.\n\nDo not manually edit observed serials, associations, firmware or check outcomes. `dpf.release` selects the release's BOM; changing it does not upgrade software. An omitted release uses the CLI's default, so record the CLI version and review `inventory.software` when comparing runs from different versions. Specify the release explicitly when you want to retain that choice.", + "transitions": [ + { + "to": "run", + "label": "Same scope ready" + }, + { + "to": "new-scope", + "label": "Scope changed" + } + ] + }, + { + "id": "run", + "title": "CLI machine: repeat reads on the selected endpoints", + "body": "Launch Kit repeats the [same reads on the selected host, DPU, BMC and optional management API](#initial-inventory-collect), then matches the returned identities and compares them with the saved inventory. Run from the CLI machine using the same YAML path for input and output and explicitly supply `--save-cluster-config`:\n\n```bash\nl8k discover --target dpf --user-config dpf-discovered.yaml \\\n --save-cluster-config dpf-discovered.yaml --output json \\\n > refresh.json 2> refresh.log\n```\n\nA complete in-place refresh retains `source-before.yaml` before replacing inventory while preserving user sections, comments and input intent. A partial run leaves the original input unchanged and writes its new observations separately; it does not create this backup. Relative file references are rebased if saving elsewhere; private files are not copied. Avoid simultaneous external edits, and use real output directories rather than symlink aliases. Credentials, trust files and configured lease files cannot be used as output destinations.\n\nWatch live source/check status on stderr: RUNNING is emitted before the operation, followed by its result. Endpoint names/addresses distinguish concurrent work. To keep progress visible with JSON, redirect stdout only (`>discover.json`). Add `--log-level debug` for stage timing or `--log-level trace` for request/tool metadata; `2>discover.log` saves these messages instead of displaying them. Quiet suppresses progress.", + "transitions": [ + { + "to": "compare", + "label": "Refresh finished" + } + ] + }, + { + "id": "compare", + "title": "CLI machine: compare new observations and reasons", + "body": "Review `diff`, current checks, `errors`, cleanup and successfully written `artifacts` in the result. Compare physical identity, firmware, relevant hardware/mode/network observations and resolved release requirements. A failed current observation stays unknown rather than inheriting an earlier pass. Reordered lists alone are not changes to the observed configuration.\n\nOn partial collection, the previous handoff remains intact and the new partial YAML/report live under `dpf-discovered.discovery//`. Inspect the new report instead of mistaking the unchanged handoff for fresh success. Required failures need correction and another run. Optional missing tools or inaccessible properties remain explained observations; they do not make deployment readiness pass.", + "transitions": [ + { + "to": "retain", + "label": "Complete refresh reviewed" + }, + { + "to": "repair", + "label": "Partial or failed refresh" + } + ] + }, + { + "id": "repair", + "title": "Endpoint owner or CLI operator: repair and retry", + "body": "Use the saved reason and affected source to repair the external prerequisite or fix the input. Preserve the unsuccessful run's report. Do not overwrite failure evidence with an old successful observation. If the cause is a real endpoint, trust or identity change, switch to a fresh seed and output.", + "transitions": [ + { + "to": "run", + "label": "Same scope corrected" + }, + { + "to": "new-scope", + "label": "Actual scope change" + } + ] + }, + { + "id": "new-scope", + "title": "CLI machine: start a new inventory", + "body": "Create a new seed from the desired access configuration, without copying old `inventory` facts, and select a new handoff path. Recheck endpoint ownership/trust and follow [initial inventory](#initial-inventory). Keep the previous estate's handoff and report as history.", + "transitions": [] + }, + { + "id": "retain", + "title": "CLI machine: retain the refreshed files", + "body": "Keep the current YAML, JSON/stdout report and stderr log, plus `source-before.yaml` when a complete in-place refresh created it. Discovery completion still means a successful inventory workflow, with deployment `acceptance: not-evaluated`. Unknown software requirements and unverified deployment prerequisites remain unresolved; a refreshed inventory does not enable DPF deployment.", + "transitions": [] + } + ] + } + ] +} diff --git a/pkg/cmd/discover.go b/pkg/cmd/discover.go index 435747af..4c2521c1 100644 --- a/pkg/cmd/discover.go +++ b/pkg/cmd/discover.go @@ -17,11 +17,13 @@ package cmd import ( + apperrors "github.com/nvidia/k8s-launch-kit/pkg/errors" "github.com/spf13/cobra" "github.com/nvidia/k8s-launch-kit/pkg/configflags" "github.com/nvidia/k8s-launch-kit/pkg/options" "github.com/nvidia/k8s-launch-kit/pkg/target" + dpftarget "github.com/nvidia/k8s-launch-kit/pkg/target/dpf" hosttarget "github.com/nvidia/k8s-launch-kit/pkg/target/host" ) @@ -40,7 +42,11 @@ NicDevice CRs, and torn down when discovery finishes. Discovery groups nodes by hardware, detects east-west vs north-south NICs, and probes OFED-dependent modules. With --user-config, it replaces -clusterConfig and discoveredSecondaryNetwork, then applies explicit CLI overrides.`, +clusterConfig and discoveredSecondaryNetwork, then applies explicit CLI overrides. + +With --target dpf, inspect the independent management/BMC estate using ordinary +YAML. The inventory preview requires discovery.probes.mode: inventory; management +probes and hardware support require qualified release assets.`, Example: ` # Basic discovery l8k discover --kubeconfig ~/.kube/config \ --save-cluster-config ./cluster-config.yaml @@ -67,6 +73,13 @@ clusterConfig and discoveredSecondaryNetwork, then applies explicit CLI override l8k discover --save-cluster-config ./cluster-config.yaml \ --output json 2>/dev/null`, Run: func(cmd *cobra.Command, args []string) { + if targetName == string(target.DPF) && len(args) > 0 { + exitWithError(apperrors.NewValidationError("discover takes no positional arguments", nil, "Use --user-config"), outputFormat) + } + if targetName == string(target.DPF) { + runTargetCommand(cmd, target.Discover, dpftarget.NewDiscoverAdapter(dpftarget.Request{UserConfig: userConfig, OutputPath: saveClusterConfig, ExplicitOutput: cmd.Flags().Changed("save-cluster-config")})) + return + } opts := mustCollectConfigFlags(cmd, discoverConfigOptions) opts.ConfigDir = configDir opts.DiscoverClusterConfig = true @@ -93,6 +106,8 @@ clusterConfig and discoveredSecondaryNetwork, then applies explicit CLI override func init() { rootCmd.AddCommand(discoverCmd) addTargetFlag(discoverCmd) + discoverCmd.Flags().BoolVarP(&quietFlag, "quiet", "q", false, "Suppress discovery progress (final result and errors still shown)") + setFlagGroup(discoverCmd, "quiet", GroupOutputLogging) discoverCmd.Flags().StringVar(&kubeconfig, "kubeconfig", "", "Path to kubeconfig file (falls back to $KUBECONFIG, then ~/.kube/config)") discoverCmd.Flags().StringVar(&userConfig, "user-config", "", "Base config to merge with discovered hardware") diff --git a/pkg/cmd/dpf_progress_test.go b/pkg/cmd/dpf_progress_test.go new file mode 100644 index 00000000..3a84249e --- /dev/null +++ b/pkg/cmd/dpf_progress_test.go @@ -0,0 +1,110 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package cmd + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +type dpfProgressBuffer struct { + mu sync.Mutex + data bytes.Buffer +} + +func (b *dpfProgressBuffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.data.Write(p) +} +func (b *dpfProgressBuffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.data.String() +} + +func TestDPFCLIProgressWhileRemoteRequestIsPending(t *testing.T) { + for _, quiet := range []bool{false, true} { + t.Run(fmt.Sprintf("quiet=%t", quiet), func(t *testing.T) { + entered, release := make(chan struct{}), make(chan struct{}) + var enteredOnce, releaseOnce sync.Once + unblock := func() { releaseOnce.Do(func() { close(release) }) } + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + enteredOnce.Do(func() { close(entered) }) + select { + case <-release: + case <-r.Context().Done(): + return + } + w.WriteHeader(http.StatusServiceUnavailable) + _, _ = w.Write([]byte(`{"message":"private-response-canary"}`)) + })) + defer server.Close() + defer unblock() + dir, pathErr := filepath.EvalSymlinks(t.TempDir()) + require.NoError(t, pathErr) + seed := filepath.Join(dir, "site.yaml") + source := fmt.Sprintf("credentials:\n reader: {username: admin, passwordRef: {env: DPF_PROGRESS_TEST_PASSWORD}}\ndiscovery:\n tls: {insecureSkipVerify: true}\n hosts: [{name: worker-dpu, bmcEndpoint: '%s', credentialRef: reader}]\n probes: {mode: inventory}\n", server.URL) + require.NoError(t, os.WriteFile(seed, []byte(source), 0600)) + ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) + defer cancel() + args := []string{"-test.run=^TestTargetCLIHelperProcess$", "--", "discover", "--target", "dpf", "--user-config", seed, "--save-cluster-config", filepath.Join(dir, "inventory.yaml"), "--output", "json"} + if quiet { + args = append(args, "--quiet") + } + command := exec.CommandContext(ctx, os.Args[0], args...) + command.Env = append(os.Environ(), "L8K_TARGET_CLI_HELPER=1", "DPF_PROGRESS_TEST_PASSWORD=password-canary") + // Separate streams are required to verify the public JSON/progress contract. + var stdout bytes.Buffer + stderr := &dpfProgressBuffer{} + command.Stdout, command.Stderr = &stdout, stderr + done := make(chan error, 1) + go func() { done <- command.Run() }() + select { + case <-entered: + case err := <-done: + t.Fatalf("CLI exited before request: %v; %s", err, stderr.String()) + case <-ctx.Done(): + t.Fatal("CLI never reached fixture") + } + if quiet { + require.Empty(t, stderr.String()) + } else { + require.Eventually(t, func() bool { return bytes.Contains([]byte(stderr.String()), []byte(`"worker-dpu"`)) }, time.Second, 10*time.Millisecond) + require.Contains(t, stderr.String(), "[RUNNING]") + require.Contains(t, stderr.String(), "Redfish connection/session") + require.NotContains(t, stderr.String(), "[PASS] Publication") + } + unblock() + err := <-done + var exit *exec.ExitError + require.ErrorAs(t, err, &exit) + require.Equal(t, 3, exit.ExitCode()) + var report map[string]any + require.NoError(t, json.Unmarshal(stdout.Bytes(), &report)) + require.Equal(t, "dpf", report["target"]) + if quiet { + require.Empty(t, stderr.String()) + } else { + require.Contains(t, stderr.String(), "[FAIL]") + require.Contains(t, stderr.String(), "HTTP_FAILED") + } + for _, secret := range []string{"password-canary", "private-response-canary"} { + require.NotContains(t, stdout.String()+stderr.String(), secret) + } + }) + } +} diff --git a/pkg/cmd/schema_test.go b/pkg/cmd/schema_test.go index 7fe663ec..c9862508 100644 --- a/pkg/cmd/schema_test.go +++ b/pkg/cmd/schema_test.go @@ -38,7 +38,7 @@ func TestSchemaDescribesTargetsAdditively(t *testing.T) { for _, phase := range target.PublicPhases() { assert.Truef(t, host.Phases[string(phase)].Available, "host phase %s", phase) - assert.Falsef(t, dpf.Phases[string(phase)].Available, "dpf phase %s", phase) + assert.Equalf(t, phase == target.Discover, dpf.Phases[string(phase)].Available, "dpf phase %s", phase) assert.NotEmptyf(t, dpf.Phases[string(phase)].Reason, "dpf phase %s reason", phase) } diff --git a/pkg/cmd/target_cli.go b/pkg/cmd/target_cli.go index d213a481..776ca8b2 100644 --- a/pkg/cmd/target_cli.go +++ b/pkg/cmd/target_cli.go @@ -30,6 +30,7 @@ import ( "github.com/nvidia/k8s-launch-kit/pkg/configflags" apperrors "github.com/nvidia/k8s-launch-kit/pkg/errors" "github.com/nvidia/k8s-launch-kit/pkg/target" + dpftarget "github.com/nvidia/k8s-launch-kit/pkg/target/dpf" hosttarget "github.com/nvidia/k8s-launch-kit/pkg/target/host" ) @@ -67,7 +68,7 @@ func addTargetFlag(cmd *cobra.Command) { &targetName, "target", string(target.Host), - "Deployment target: host (default) or dpf (reserved; phases are unavailable until the DPF driver is added)", + "Deployment target: host (default) or dpf (experimental inventory discovery)", ) setFlagGroup(cmd, "target", GroupTarget) setFlagTargetScope(cmd, []target.Name{target.Host, target.DPF}, "target") @@ -166,27 +167,13 @@ func (d unavailableTargetDriver) Bind(invocation target.Invocation) (target.Oper } } -func dpfTargetDescriptor() target.Descriptor { - const reason = "the DPF target name and CLI boundary are reserved, but the DPF driver is not available in this build" - unavailable := target.Capability{Available: false, Reason: reason} - return target.Descriptor{ - Name: target.DPF, - Description: "DPU-plane provisioning through NVIDIA DPF Operator", - Phases: map[target.Phase]target.Capability{ - target.Discover: unavailable, - target.Generate: unavailable, - target.Deploy: unavailable, - target.Validate: unavailable, - target.Pipeline: unavailable, - }, - } -} +func dpfTargetDescriptor() target.Descriptor { return dpftarget.Descriptor() } func targetDescriptors() []target.Descriptor { return []target.Descriptor{hosttarget.Descriptor(), dpfTargetDescriptor()} } -func bindTargetCommand(cmd *cobra.Command, phase target.Phase, hostDriver target.Driver) (target.Operation, error) { +func bindTargetCommand(cmd *cobra.Command, phase target.Phase, driver target.Driver) (target.Operation, error) { selected, err := target.ParseName(targetName) if err != nil { return nil, err @@ -195,9 +182,19 @@ func bindTargetCommand(cmd *cobra.Command, phase target.Phase, hostDriver target return nil, err } + hostDriver := target.Driver(unavailableTargetDriver{descriptor: hosttarget.Descriptor()}) + dpfDriver := target.Driver(unavailableTargetDriver{descriptor: dpfTargetDescriptor()}) + switch driver.Descriptor().Name { + case target.Host: + hostDriver = driver + case target.DPF: + dpfDriver = driver + default: + return nil, fmt.Errorf("unexpected driver") + } registry, err := target.NewRegistry( hostDriver, - unavailableTargetDriver{descriptor: dpfTargetDescriptor()}, + dpfDriver, ) if err != nil { return nil, fmt.Errorf("construct target registry: %w", err) @@ -225,8 +222,8 @@ func bindTargetCommand(cmd *cobra.Command, phase target.Phase, hostDriver target }) } -func runTargetCommand(cmd *cobra.Command, phase target.Phase, hostDriver target.Driver) { - operation, err := bindTargetCommand(cmd, phase, hostDriver) +func runTargetCommand(cmd *cobra.Command, phase target.Phase, driver target.Driver) { + operation, err := bindTargetCommand(cmd, phase, driver) if err != nil { var structured *apperrors.StructuredError if errors.As(err, &structured) { @@ -292,6 +289,7 @@ func markDiscoverTargetScopes() { "topology-scheme", "ip-version", "topology-file", "spectrum-x-config", "spectrum-x-configmap-name") markConfigFlagTargetScopes(discoverCmd, configflags.ScopeDiscover) + setFlagTargetScope(discoverCmd, []target.Name{target.Host, target.DPF}, "user-config", "save-cluster-config", "quiet") } func markGenerateTargetScopes() { diff --git a/pkg/cmd/target_cli_test.go b/pkg/cmd/target_cli_test.go index 7220eade..c80b9099 100644 --- a/pkg/cmd/target_cli_test.go +++ b/pkg/cmd/target_cli_test.go @@ -250,7 +250,7 @@ func TestTargetDescriptors(t *testing.T) { assert.Equal(t, target.DPF, descriptors[1].Name) for _, phase := range target.PublicPhases() { assert.True(t, descriptors[0].Capability(phase).Available) - assert.False(t, descriptors[1].Capability(phase).Available) + assert.Equal(t, phase == target.Discover, descriptors[1].Capability(phase).Available) assert.NotEmpty(t, descriptors[1].Capability(phase).Reason) } } @@ -291,6 +291,9 @@ func TestTargetCLIProcess(t *testing.T) { }) for _, phase := range target.PublicPhases() { + if phase == target.Discover { + continue + } t.Run("dpf "+string(phase)+" capability error", func(t *testing.T) { output, exitCode := runCLIHelper(t, string(phase), "--target", "dpf") assert.Equal(t, 2, exitCode) @@ -314,8 +317,8 @@ func TestTargetCLIProcess(t *testing.T) { t.Run("dpf capability errors preserve json output mode", func(t *testing.T) { output, exitCode := runCLIHelper(t, "discover", "--target", "dpf", "--output", "json") assert.Equal(t, 2, exitCode) - assert.Contains(t, output, `"code": "VALIDATION_ERROR"`) - assert.Contains(t, output, `"success": false`) + assert.Contains(t, output, `"code":"INVALID_CONFIGURATION"`) + assert.Contains(t, output, `"target":"dpf"`) }) t.Run("unknown target is distinct from unavailable phase", func(t *testing.T) { diff --git a/pkg/config/yaml_patch.go b/pkg/config/yaml_patch.go index 6a6fa12f..a96433c3 100644 --- a/pkg/config/yaml_patch.go +++ b/pkg/config/yaml_patch.go @@ -10,6 +10,7 @@ import ( "reflect" "strings" + "github.com/nvidia/k8s-launch-kit/pkg/yamlinput" yaml3 "gopkg.in/yaml.v3" ) @@ -17,68 +18,8 @@ import ( // expanded. YAML merges are shallow: an explicit mapping replaces the entire // inherited mapping at that key. Earlier mappings in a merge sequence win. // Sharing this expansion keeps presence tracking and source patches consistent. -func expandYAMLNode(node *yaml3.Node, active map[*yaml3.Node]bool) (*yaml3.Node, error) { - if node == nil { - return nil, fmt.Errorf("missing YAML node") - } - if active[node] { - return nil, fmt.Errorf("cyclic YAML alias") - } - active[node] = true - defer delete(active, node) - if node.Kind == yaml3.AliasNode { - return expandYAMLNode(node.Alias, active) - } - copy := *node - copy.Anchor = "" - copy.Content = nil - for _, child := range node.Content { - expanded, err := expandYAMLNode(child, active) - if err != nil { - return nil, err - } - copy.Content = append(copy.Content, expanded) - } - if copy.Kind != yaml3.MappingNode { - return ©, nil - } - entries := copy.Content - copy.Content = nil - for i := 0; i+1 < len(entries); i += 2 { - if entries[i].Tag != "!!merge" { - continue - } - mappings := []*yaml3.Node{entries[i+1]} - if entries[i+1].Kind == yaml3.SequenceNode { - mappings = entries[i+1].Content - } - for _, mapping := range mappings { - if mapping.Kind != yaml3.MappingNode { - return nil, fmt.Errorf("YAML merge value must be a mapping or sequence of mappings") - } - for j := 0; j+1 < len(mapping.Content); j += 2 { - putYAMLField(©, mapping.Content[j], mapping.Content[j+1], false) - } - } - } - for i := 0; i+1 < len(entries); i += 2 { - if entries[i].Tag != "!!merge" { - putYAMLField(©, entries[i], entries[i+1], true) - } - } - return ©, nil -} - -func putYAMLField(mapping, key, value *yaml3.Node, replace bool) { - for i := 0; i+1 < len(mapping.Content); i += 2 { - if mapping.Content[i].Value == key.Value { - if replace { - mapping.Content[i], mapping.Content[i+1] = key, value - } - return - } - } - mapping.Content = append(mapping.Content, key, value) +func expandYAMLNode(node *yaml3.Node, _ map[*yaml3.Node]bool) (*yaml3.Node, error) { + return yamlinput.Expand(node) } // PatchConfigYAML updates only the requested YAML paths, preserving all other diff --git a/pkg/dpf/addresslookup/lookup.go b/pkg/dpf/addresslookup/lookup.go new file mode 100644 index 00000000..74182c59 --- /dev/null +++ b/pkg/dpf/addresslookup/lookup.go @@ -0,0 +1,196 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package addresslookup resolves explicitly selected MAC locators without contacting candidates. +package addresslookup + +import ( + "fmt" + "net" + "net/netip" + "slices" + "strconv" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +type Source struct { + File string `yaml:"file"` + Format string `yaml:"format"` +} +type Selector struct { + Name string `yaml:"name"` + Role string `yaml:"role"` + MAC string `yaml:"mac"` +} +type Config struct { + Sources []Source `yaml:"sources,omitempty"` + Devices []Selector `yaml:"devices"` +} + +// Request contains normalized selectors and absolute source paths, no credentials. +type Request struct { + Sources []Source + Devices []Selector +} +type Neighbor struct{ Address, MAC, State, Source, EvidenceRef string } + +func MAC(value string) (string, error) { + m, err := net.ParseMAC(value) + if err != nil || len(m) != 6 || m[0]&1 != 0 || m.String() == "00:00:00:00:00:00" { + return "", fmt.Errorf("expected a unicast six-byte MAC address") + } + return m.String(), nil +} +func Validate(c Config) (Request, error) { + r := Request{Sources: slices.Clone(c.Sources), Devices: slices.Clone(c.Devices)} + if len(r.Devices) == 0 || len(r.Devices) > 256 || len(r.Sources) > 16 { + return r, fmt.Errorf("addressLookup needs 1-256 devices and at most 16 sources") + } + names := map[string]bool{} + for i, d := range r.Devices { + if d.Name == "" || len(d.Name) > 63 || strings.ContainsAny(d.Name, "/\\\n\r\t ") || names[d.Name] { + return r, fmt.Errorf("addressLookup device names must be unique nonempty labels") + } + names[d.Name] = true + if d.Role != "arm" && d.Role != "bmc" { + return r, fmt.Errorf("addressLookup device role must be arm or bmc") + } + mac, err := MAC(d.MAC) + if err != nil { + return r, err + } + r.Devices[i].MAC = mac + } + for _, s := range r.Sources { + if s.File == "" || s.Format != "dnsmasq-leases" { + return r, fmt.Errorf("addressLookup source needs file and format: dnsmasq-leases") + } + } + return r, nil +} + +type lease struct { + mac, address string + expired bool +} + +func leases(data []byte, now time.Time) ([]lease, error) { + if len(data) > 1<<20 { + return nil, fmt.Errorf("lease source exceeds 1 MiB") + } + out := []lease{} + for _, line := range strings.Split(string(data), "\n") { + f := strings.Fields(line) + if len(f) == 0 || strings.HasPrefix(f[0], "#") { + continue + } + // The five-column dnsmasq IPv4 lease format; never guess other formats. + if len(f) != 5 { + return nil, fmt.Errorf("invalid dnsmasq lease record") + } + expiry, err := strconv.ParseInt(f[0], 10, 64) + if err != nil || expiry < 0 { + return nil, fmt.Errorf("invalid lease expiry") + } + mac, err := MAC(f[1]) + if err != nil { + return nil, err + } + ip, err := netip.ParseAddr(f[2]) + if err != nil || !ip.Is4() || !ip.IsGlobalUnicast() { + return nil, fmt.Errorf("invalid lease address") + } + out = append(out, lease{mac, ip.String(), expiry != 0 && expiry <= now.Unix()}) + if len(out) > 4096 { + return nil, fmt.Errorf("too many lease records") + } + } + return out, nil +} + +// Collect uses a bounded reader supplied by the caller. Invalid sources contribute +// no candidates. Valid candidates remain hints even when other sources fail. +func Collect(r Request, now time.Time, read func(string, int64) ([]byte, error), neighbors []Neighbor) (*inventory.AddressLookup, map[string]inventory.Evidence) { + out := &inventory.AddressLookup{Candidates: []inventory.AddressCandidate{}, Observations: map[string]inventory.Observation{}} + evidence := map[string]inventory.Evidence{} + expired := map[string]bool{} + add := func(mac, address, source, ref string) { + for _, d := range r.Devices { + if d.MAC == mac { + out.Candidates = append(out.Candidates, inventory.AddressCandidate{Name: d.Name, Role: d.Role, MAC: mac, Address: address, Source: source, EvidenceRef: ref}) + } + } + } + for i, s := range r.Sources { + key := fmt.Sprintf("source.%d", i) + data, err := read(s.File, 1<<20) + if err != nil { + out.Observations[key] = inventory.Observation{State: "unavailable", ReasonCode: "ADDRESS_SOURCE_UNREADABLE", Message: "Cannot read the configured bounded lease file; check that it exists and is readable."} + continue + } + rows, err := leases(data, now) + if err != nil { + out.Observations[key] = inventory.Observation{State: "unavailable", ReasonCode: "ADDRESS_SOURCE_INVALID", Message: "The lease file is not a bounded supported dnsmasq IPv4 lease table."} + continue + } + ref := inventory.Digest("dpf-address-source-v1", []string{key, inventory.Digest("lease-bytes-v1", data)}) + evidence[ref] = inventory.Evidence{Kind: "lease-file", InputRef: fmt.Sprintf("/discovery/addressLookup/sources/%d", i), ObservedAt: now, Vantage: "executor", Property: "MAC/IP/expiry"} + out.Observations[key] = inventory.Observation{State: "observed"} + for _, row := range rows { + if row.expired { + expired[row.mac] = true + continue + } + add(row.mac, row.address, key, ref) + } + } + for _, n := range neighbors { + mac, err := MAC(n.MAC) + if err != nil { + continue + } + ip, err := netip.ParseAddr(n.Address) + if err != nil || !ip.IsGlobalUnicast() { + continue + } + // Only usable kernel entries; stale entries are not proof of current reachability. + if n.State != "REACHABLE" && n.State != "PERMANENT" { + continue + } + add(mac, ip.String(), n.Source, n.EvidenceRef) + } + slices.SortFunc(out.Candidates, func(a, b inventory.AddressCandidate) int { + return strings.Compare(a.Name+"/"+a.Address+"/"+a.Source, b.Name+"/"+b.Address+"/"+b.Source) + }) + out.Candidates = slices.Compact(out.Candidates) + for _, d := range r.Devices { + ips := map[string]bool{} + for _, c := range out.Candidates { + if c.Name == d.Name { + ips[c.Address] = true + } + } + o := inventory.Observation{State: "unknown", ReasonCode: "ADDRESS_NOT_FOUND", Message: "No usable lease or reachable neighbor matches the requested MAC; provide an address or refresh the source."} + if expired[d.MAC] { + o.ReasonCode = "ADDRESS_LEASE_EXPIRED" + o.Message = "Only expired matching lease evidence is available; refresh the lease source." + } + if len(ips) > 1 { + o.ReasonCode = "ADDRESS_AMBIGUOUS" + o.Message = "Sources report multiple addresses for this MAC; verify the intended endpoint before connecting." + } + if len(ips) == 1 { + o.State = "candidate" + o.ReasonCode = "ADDRESS_UNVERIFIED" + o.Message = "One address candidate was found; verify endpoint identity and configure explicit credentials and trust before connecting." + for ip := range ips { + o.Values = []string{ip} + } + } + out.Observations["device."+d.Name] = o + } + return out, evidence +} diff --git a/pkg/dpf/addresslookup/lookup_test.go b/pkg/dpf/addresslookup/lookup_test.go new file mode 100644 index 00000000..ab273f5a --- /dev/null +++ b/pkg/dpf/addresslookup/lookup_test.go @@ -0,0 +1,81 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package addresslookup + +import ( + "fmt" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "testing" + "time" +) + +func TestLookup(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "DPF address candidates") } + +var _ = Describe("Explicit MAC address lookup", func() { + var r Request + BeforeEach(func() { + var err error + r, err = Validate(Config{Sources: []Source{{File: "leases", Format: "dnsmasq-leases"}}, Devices: []Selector{{Name: "dpu1", Role: "arm", MAC: "02:00:00:00:00:01"}}}) + Expect(err).NotTo(HaveOccurred()) + }) + It("retains usable leases as candidates with provenance, not verified endpoints", func() { + out, ev := Collect(r, time.Unix(100, 0), func(string, int64) ([]byte, error) { return []byte("200 02:00:00:00:00:01 192.0.2.3 ignored *\n"), nil }, nil) + Expect(out.Candidates).To(HaveLen(1)) + Expect(ev).To(HaveLen(1)) + Expect(out.Observations["device.dpu1"].State).To(Equal("candidate")) + Expect(out.Candidates[0].EvidenceRef).NotTo(BeEmpty()) + Expect(out.Observations["source.0"].Values).To(BeEmpty(), "source-byte digests remain in the evidence map and candidate provenance, not observation facts") + }) + It("keeps source-byte evidence changes out of address facts", func() { + first, firstEvidence := Collect(r, time.Unix(100, 0), func(string, int64) ([]byte, error) { + return []byte("200 02:00:00:00:00:01 192.0.2.3 host-a *\n"), nil + }, nil) + second, secondEvidence := Collect(r, time.Unix(100, 0), func(string, int64) ([]byte, error) { + return []byte("200 02:00:00:00:00:01 192.0.2.3 host-b *\n"), nil + }, nil) + Expect(first.Candidates).To(HaveLen(1)) + Expect(second.Candidates).To(HaveLen(1)) + Expect(first.Candidates[0].Address).To(Equal(second.Candidates[0].Address)) + Expect(first.Candidates[0].EvidenceRef).NotTo(Equal(second.Candidates[0].EvidenceRef)) + Expect(firstEvidence).NotTo(BeEmpty()) + Expect(secondEvidence).NotTo(BeEmpty()) + Expect(first.Observations["source.0"].Values).To(BeEmpty()) + Expect(second.Observations["source.0"].Values).To(BeEmpty()) + }) + It("reports expired, missing, unreadable and malformed sources distinctly", func() { + for _, tc := range []struct { + data, code string + err error + }{{"99 02:00:00:00:00:01 192.0.2.3 * *", "ADDRESS_LEASE_EXPIRED", nil}, {"", "ADDRESS_NOT_FOUND", nil}, {"", "ADDRESS_SOURCE_UNREADABLE", fmt.Errorf("secret-bearing failure")}, {"bad data", "ADDRESS_SOURCE_INVALID", nil}} { + out, _ := Collect(r, time.Unix(100, 0), func(string, int64) ([]byte, error) { return []byte(tc.data), tc.err }, nil) + Expect(out.Candidates).To(BeEmpty()) + key := "device.dpu1" + if tc.code == "ADDRESS_SOURCE_UNREADABLE" || tc.code == "ADDRESS_SOURCE_INVALID" { + key = "source.0" + } + Expect(out.Observations[key].ReasonCode).To(Equal(tc.code)) + Expect(out.Observations[key].Message).NotTo(BeEmpty()) + Expect(out.Observations[key].Message).NotTo(ContainSubstring("secret-bearing")) + } + }) + It("retains disagreements and does not select an address", func() { + out, _ := Collect(r, time.Unix(100, 0), func(string, int64) ([]byte, error) { return []byte("200 02:00:00:00:00:01 192.0.2.3 * *"), nil }, []Neighbor{{Address: "192.0.2.4", MAC: r.Devices[0].MAC, State: "REACHABLE", Source: "host", EvidenceRef: "public"}}) + Expect(out.Candidates).To(HaveLen(2)) + Expect(out.Observations["device.dpu1"].ReasonCode).To(Equal("ADDRESS_AMBIGUOUS")) + Expect(out.Observations["device.dpu1"].Values).To(BeEmpty()) + }) + It("rejects unsupported sources and invalid selectors", func() { + for _, c := range []Config{{}, {Devices: []Selector{{Name: "x", Role: "arm", MAC: "ff:ff:ff:ff:ff:ff"}}}, {Devices: []Selector{{Name: "x", Role: "host", MAC: "02:00:00:00:00:01"}}}, {Sources: []Source{{File: "x", Format: "guessed"}}, Devices: r.Devices}} { + _, err := Validate(c) + Expect(err).To(HaveOccurred()) + } + }) + It("ignores stale neighbors and invalidates the whole malformed lease source", func() { + out, _ := Collect(r, time.Unix(100, 0), func(string, int64) ([]byte, error) { + return []byte("200 02:00:00:00:00:01 192.0.2.3 * *\nmalformed"), nil + }, []Neighbor{{Address: "192.0.2.4", MAC: r.Devices[0].MAC, State: "STALE", Source: "host"}}) + Expect(out.Candidates).To(BeEmpty()) + Expect(out.Observations["source.0"].ReasonCode).To(Equal("ADDRESS_SOURCE_INVALID")) + }) +}) diff --git a/pkg/dpf/artifacts/compact_test.go b/pkg/dpf/artifacts/compact_test.go new file mode 100644 index 00000000..60558423 --- /dev/null +++ b/pkg/dpf/artifacts/compact_test.go @@ -0,0 +1,139 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package artifacts + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +var _ = Describe("compact device artifact handoff", func() { + It("publishes diagnostics only while preserving consumer facts and comparisons", func() { + for _, complete := range []bool{true, false} { + dir, err := filepath.EvalSymlinks(GinkgoT().TempDir()) + Expect(err).NotTo(HaveOccurred()) + seed := filepath.Join(dir, "site.yaml") + Expect(os.WriteFile(seed, []byte("credentials: {}\ndiscovery: {hosts: [], dpus: []}\n"), 0600)).To(Succeed()) + input, err := config.Load(seed) + Expect(err).NotTo(HaveOccurred()) + software, err := releases.Resolve("26.4.0") + Expect(err).NotTo(HaveOccurred()) + inv := &inventory.Inventory{ + FormatVersion: inventory.FormatVersion, Software: &software, + Collection: inventory.Collection{Mode: "inventory"}, + RequestedScope: []inventory.ScopeEndpoint{{Role: "dpus", Endpoint: "https://dpu.invalid", TLSTrustDigest: "custom-ca-test"}}, + DPUs: []inventory.Device{{ + ID: "dpu-one", RecordKey: "dpu-source", BMCEndpoint: "https://dpu.invalid", SerialNumber: "CARDONE", Model: "B3210", ObservationState: "observed", + Observations: map[string]inventory.Observation{ + "bmc-administrator": {State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED", Tool: "redfish"}, + "firmware-inventory": {State: "observed"}, + "managers": {State: "observed"}, + "bmc-ntp": {State: "unavailable", ReasonCode: "ACCESS_DENIED", Message: "NTP read denied."}, + "missing-fact": {State: "unknown", ReasonCode: "PROPERTY_NOT_EXPOSED", Message: "Requested property is absent."}, + "client-certificate-support": {State: "unknown", ReasonCode: "NOT_EVALUATED", Message: "Deferred to provisioning."}, + }, + Firmware: []inventory.Firmware{{Component: "nic", State: "known", RawVersion: "32.38.1002", Activation: "running", ResourceURI: "/redfish/v1/UpdateService/FirmwareInventory/NIC"}}, + }}, + Hosts: []inventory.Device{{ID: "host-one", BMCEndpoint: "https://host.invalid", RecordKey: "host-source", SystemUUID: "00000000-0000-0000-0000-000000000001", ObservationState: "observed", Adapters: []inventory.Adapter{{Key: "slot-one", IdentityState: "known", SerialNumber: "CARDONE"}}}}, + Associations: []inventory.Association{{DPUID: "dpu-one", HostID: "host-one", HostAdapterKey: "slot-one", SerialNumber: "CARDONE"}}, + } + beforeChecks := discovery.SoftwareChecks(inv) + Expect(beforeChecks).To(ContainElement(And(HaveField("ID", "dpf.software.nic"), HaveField("State", "passed")))) + outcome := "complete" + if !complete { + outcome = "partial" + // Identity can already exist when a final recheck fails. Never + // infer success from that ID when reading the partial result. + inv.Hosts[0].ObservationState = "unavailable" + inv.Hosts[0].ReasonCode = "IDENTITY_CHANGED" + beforeChecks = discovery.SoftwareChecks(inv) + } + discovery.Canonicalize(inv) + r := &inventory.Report{SchemaVersion: inventory.FormatVersion, Inventory: inv, Discovery: inventory.Discovery{Outcome: outcome, Checks: beforeChecks}} + store, err := New(input, filepath.Join(dir, "discovered.yaml"), true) + Expect(err).NotTo(HaveOccurred()) + Expect(store.Publish(context.Background(), r)).To(Succeed()) + path := store.Output + if !complete { + path = filepath.Join(store.RunDir, "partial.yaml") + } + data, err := os.ReadFile(path) + Expect(err).NotTo(HaveOccurred()) + var wire map[string]any + Expect(yaml.Unmarshal(data, &wire)).To(Succeed()) + assertCompactDeviceWire(wire["inventory"].(map[string]any), complete) + loaded, err := config.Load(path) + Expect(err).NotTo(HaveOccurred()) + Expect(checkResults(discovery.SoftwareChecks(loaded.Site.Inventory))).To(Equal(checkResults(beforeChecks))) + Expect(discovery.Diff(inv, loaded.Site.Inventory)).To(BeEmpty()) + Expect(loaded.Site.Inventory.RequestedScope).To(BeNil()) + discovery.Canonicalize(loaded.Site.Inventory) + Expect(loaded.Site.Inventory.ScopeDigest).To(Equal(inv.ScopeDigest)) + Expect(loaded.Site.Inventory.Fingerprint).To(Equal(inv.Fingerprint)) + Expect(loaded.Site.Inventory.DPUs[0].Observations["bmc-administrator"].ReasonCode).To(Equal("BMC_ADMIN_VERIFIED")) + data, err = os.ReadFile(filepath.Join(store.RunDir, "report.json")) + Expect(err).NotTo(HaveOccurred()) + Expect(json.Unmarshal(data, &wire)).To(Succeed()) + assertCompactDeviceWire(wire["inventory"].(map[string]any), complete) + var report inventory.Report + Expect(json.Unmarshal(data, &report)).To(Succeed()) + Expect(checkResults(discovery.SoftwareChecks(report.Inventory))).To(Equal(checkResults(beforeChecks))) + Expect(discovery.Diff(inv, report.Inventory)).To(BeEmpty()) + beforeJSON, err := json.Marshal(beforeChecks) + Expect(err).NotTo(HaveOccurred()) + afterJSON, err := json.Marshal(report.Discovery.Checks) + Expect(err).NotTo(HaveOccurred()) + Expect(afterJSON).To(MatchJSON(beforeJSON)) + // Publication is a projection, never a mutation of the collector data. + Expect(inv.DPUs[0].ObservationState).To(Equal("observed")) + Expect(inv.DPUs[0].Observations).To(HaveKey("managers")) + } + }) +}) + +func assertCompactDeviceWire(inv map[string]any, complete bool) { + dpu := inv["dpus"].([]any)[0].(map[string]any) + Expect(dpu).NotTo(HaveKey("observationState")) + Expect(dpu["bmcAdministratorVerified"]).To(BeTrue()) + Expect(dpu).NotTo(HaveKey("observations")) + issues := dpu["issues"].([]any) + Expect(issues).To(HaveLen(1)) + Expect(issues[0].(map[string]any)["field"]).To(Equal("missing-fact")) + Expect(dpu["firmware"].(map[string]any)["nic"]).To(HaveLen(1)) + software := inv["software"].(map[string]any) + Expect(software).NotTo(HaveKey("sources")) + hardware := software["hardware"].([]any)[0].(map[string]any) + Expect(hardware).NotTo(HaveKey("requirements")) + Expect(hardware["minimumNICFirmware"]).To(Equal("32.38.1002")) + host := inv["hosts"].([]any)[0].(map[string]any) + Expect(host).NotTo(HaveKey("firmware")) + Expect(host).NotTo(HaveKey("time")) + if complete { + Expect(host).NotTo(HaveKey("observationState")) + } else { + Expect(host).NotTo(HaveKey("id")) + Expect(host["issues"].([]any)[0].(map[string]any)["code"]).To(Equal("IDENTITY_CHANGED")) + } +} + +// Provenance prose and source references are not part of the typed handoff; +// check identities, applicability, severity and reason codes remain unchanged. +func checkResults(checks []inventory.Check) []inventory.Check { + result := append([]inventory.Check(nil), checks...) + for i := range result { + result[i].Message = "" + result[i].EvidenceRefs = nil + } + return result +} diff --git a/pkg/dpf/artifacts/replace_other.go b/pkg/dpf/artifacts/replace_other.go new file mode 100644 index 00000000..d1554281 --- /dev/null +++ b/pkg/dpf/artifacts/replace_other.go @@ -0,0 +1,9 @@ +//go:build !windows + +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package artifacts + +import "os" + +func replaceFile(source, dest string) error { return os.Rename(source, dest) } diff --git a/pkg/dpf/artifacts/replace_windows.go b/pkg/dpf/artifacts/replace_windows.go new file mode 100644 index 00000000..d7df0ca3 --- /dev/null +++ b/pkg/dpf/artifacts/replace_windows.go @@ -0,0 +1,19 @@ +//go:build windows + +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package artifacts + +import "golang.org/x/sys/windows" + +func replaceFile(source, dest string) error { + s, err := windows.UTF16PtrFromString(source) + if err != nil { + return err + } + d, err := windows.UTF16PtrFromString(dest) + if err != nil { + return err + } + return windows.MoveFileEx(s, d, windows.MOVEFILE_REPLACE_EXISTING|windows.MOVEFILE_WRITE_THROUGH) +} diff --git a/pkg/dpf/artifacts/store.go b/pkg/dpf/artifacts/store.go new file mode 100644 index 00000000..596acd1c --- /dev/null +++ b/pkg/dpf/artifacts/store.go @@ -0,0 +1,334 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package artifacts publishes secret-free discovery files on local filesystems. +package artifacts + +import ( + "bytes" + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "runtime" + "slices" + "strings" + "time" + + "github.com/gofrs/flock" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "k8s.io/client-go/tools/clientcmd" +) + +type Snapshot struct { + Exists bool + Data []byte +} +type Store struct { + Output string + RunDir string + Input config.Input + Initial Snapshot + PriorInventory *inventory.Inventory +} + +func New(in config.Input, destination string, explicit bool) (*Store, error) { + if destination == "" { + destination = filepath.Join(filepath.Dir(in.Path), "dpf-discovered.yaml") + } + output, err := filepath.Abs(destination) + if err != nil { + return nil, err + } + input, err := filepath.Abs(in.Path) + if err != nil { + return nil, err + } + if input == output && !explicit { + return nil, fmt.Errorf("in-place refresh requires explicit --save-cluster-config") + } + if err := safePath(output); err != nil { + return nil, err + } + initial, err := snapshot(output) + if err != nil { + return nil, err + } + if initial.Exists && output != input { + old, e := config.Decode(initial.Data, output) + if e != nil || old.Site.Inventory == nil || !sameRequested(old.Site, in.Site) { + return nil, fmt.Errorf("output exists and is not a DPF inventory for the same requested scope") + } + } + // Protect source and referenced private files, including aliases via symlinks/hardlinks. + sourceInfo, err := os.Stat(input) + if err != nil { + return nil, fmt.Errorf("cannot inspect source") + } + outputInfo, _ := os.Stat(output) + if outputInfo != nil && os.SameFile(sourceInfo, outputInfo) && input != output { + return nil, fmt.Errorf("output aliases input") + } + protected := []string{in.Site.Management.KubeconfigRef.File} + if in.Site.Discovery.AddressLookup != nil { + for _, source := range in.Site.Discovery.AddressLookup.Sources { + protected = append(protected, source.File) + } + } + for _, c := range in.Site.Credentials { + if c.PrivateKeyRef != nil { + protected = append(protected, c.PrivateKeyRef.File) + } + if c.PasswordRef.File != nil { + protected = append(protected, *c.PasswordRef.File) + } + } + if t := in.Site.Discovery.TLS; t != nil && t.CAFile != nil { + protected = append(protected, *t.CAFile) + } + for _, e := range append(append([]config.Endpoint{}, in.Site.Discovery.DPUs...), in.Site.Discovery.Hosts...) { + if e.SSH != nil { + protected = append(protected, e.SSH.KnownHostsFile) + } + if e.TLS != nil && e.TLS.CAFile != nil { + protected = append(protected, *e.TLS.CAFile) + } + } + kubeconfigPath := in.Site.Management.KubeconfigRef.File + if !filepath.IsAbs(kubeconfigPath) { + kubeconfigPath = filepath.Join(filepath.Dir(input), kubeconfigPath) + } + if rawBytes, readErr := config.ReadReference(kubeconfigPath, 16<<20); readErr == nil { + if raw, err := clientcmd.Load(rawBytes); err == nil { + for _, auth := range raw.AuthInfos { + for _, p := range []string{auth.ClientKey, auth.ClientCertificate, auth.TokenFile} { + if p != "" { + if !filepath.IsAbs(p) { + p = filepath.Join(filepath.Dir(kubeconfigPath), p) + } + protected = append(protected, p) + } + } + } + for _, cluster := range raw.Clusters { + p := cluster.CertificateAuthority + if p != "" { + if !filepath.IsAbs(p) { + p = filepath.Join(filepath.Dir(kubeconfigPath), p) + } + protected = append(protected, p) + } + } + } + } + for _, p := range protected { + if p == "" { + continue + } + if !filepath.IsAbs(p) { + p = filepath.Join(filepath.Dir(input), p) + } + abs, _ := filepath.Abs(p) + st, _ := os.Stat(abs) + if abs == output || outputInfo != nil && st != nil && os.SameFile(st, outputInfo) { + return nil, fmt.Errorf("output aliases a private reference") + } + } + entropy := make([]byte, 16) + if _, err := rand.Read(entropy); err != nil { + return nil, err + } + runID := time.Now().UTC().Format("20060102T150405Z") + "-" + hex.EncodeToString(entropy) + runDir := filepath.Join(filepath.Dir(output), trimExtension(filepath.Base(output))+".discovery", runID) + if err := safePath(filepath.Join(runDir, "reserved")); err != nil { + return nil, err + } + if err := os.MkdirAll(runDir, 0700); err != nil { + return nil, err + } + store := &Store{Output: output, RunDir: runDir, Input: in, Initial: initial} + if initial.Exists { + old, e := config.Decode(initial.Data, output) + if e == nil { + store.PriorInventory = old.Site.Inventory + } + } + return store, nil +} +func (s *Store) RunID() string { return filepath.Base(s.RunDir) } +func trimExtension(s string) string { return s[:len(s)-len(filepath.Ext(s))] } +func sameRequested(a, b config.Site) bool { + key := func(s config.Site) string { + targets := []inventory.ScopeEndpoint{} + for _, group := range []struct { + role string + eps []config.Endpoint + }{{"dpus", s.Discovery.DPUs}, {"hosts", s.Discovery.Hosts}} { + for _, e := range group.eps { + origin, err := config.Origin(e.BMCEndpoint) + if e.SSH != nil { + origin, err = config.SSHOrigin(e.SSH.Address) + } + if err != nil { + return "" + } + selector := "" + if e.SystemURI != nil { + selector = *e.SystemURI + } + scope := inventory.ScopeEndpoint{Role: group.role, Endpoint: origin, SystemURI: selector} + if e.SSH != nil { + scope.SSHSudo = e.SSH.Sudo != nil && *e.SSH.Sudo + } + targets = append(targets, scope) + } + } + slices.SortFunc(targets, func(a, b inventory.ScopeEndpoint) int { + return strings.Compare(a.Role+a.Endpoint+a.SystemURI, b.Role+b.Endpoint+b.SystemURI) + }) + return inventory.Digest("requested", targets) + } + return key(a) == key(b) +} +func safePath(p string) error { + for current := p; ; current = filepath.Dir(current) { + st, err := os.Lstat(current) + if err == nil { + if st.Mode()&os.ModeSymlink != 0 { + return fmt.Errorf("output path contains a symlink") + } + if current == p && !st.Mode().IsRegular() { + return fmt.Errorf("output must be a regular file") + } + } else if !os.IsNotExist(err) { + return fmt.Errorf("cannot inspect output path") + } + if filepath.Dir(current) == current { + break + } + } + return nil +} +func snapshot(p string) (Snapshot, error) { + st, err := os.Stat(p) + if os.IsNotExist(err) { + return Snapshot{}, nil + } + if err != nil || !st.Mode().IsRegular() || st.Size() > 16<<20 { + return Snapshot{}, fmt.Errorf("cannot inspect bounded output file") + } + b, err := os.ReadFile(p) + return Snapshot{Exists: true, Data: b}, err +} +func (s *Store) Publish(ctx context.Context, report *inventory.Report) error { + if report == nil { + return fmt.Errorf("nil report") + } + lockPath := s.Output + ".lock" + if err := safePath(lockPath); err != nil { + return err + } + lock := flock.New(lockPath) + acquired, err := lock.TryLockContext(ctx, 50*time.Millisecond) + if err != nil || !acquired { + return fmt.Errorf("cannot acquire discovery output lock") + } + defer lock.Unlock() + source, err := os.ReadFile(s.Input.Path) + if err != nil || !bytes.Equal(source, s.Input.Source) { + return fmt.Errorf("source changed during discovery") + } + current, err := snapshot(s.Output) + if err != nil || current.Exists != s.Initial.Exists || !bytes.Equal(current.Data, s.Initial.Data) { + return fmt.Errorf("output changed during discovery") + } + if err := safePath(s.Output); err != nil { + return err + } + if report.Inventory != nil && report.Discovery.Outcome == "complete" && s.PriorInventory != nil { + if inventory.LegacyHostCardBindings(s.PriorInventory) { + return fmt.Errorf("existing destination uses legacy Redfish card bindings; verify endpoint ownership and use a fresh seed without inventory and a new output") + } + if s.PriorInventory.ScopeDigest != "" && s.PriorInventory.ScopeDigest != report.Inventory.ScopeDigest || inventory.IdentityConflict(s.PriorInventory, report.Inventory) { + return fmt.Errorf("existing destination inventory conflicts with resolved scope or physical identity") + } + } + if report.Inventory != nil { + dest := s.Output + if report.Discovery.Outcome != "complete" { + dest = filepath.Join(s.RunDir, "partial.yaml") + } + encoded, err := config.Saved(s.Input, report.Inventory, dest) + if err != nil { + return err + } + if dest == s.Input.Path { + backup := filepath.Join(s.RunDir, "source-before.yaml") + if err := AtomicWrite(backup, s.Input.Source); err != nil { + return err + } + report.Artifacts = append(report.Artifacts, inventory.Artifact{Path: backup, Kind: "source-backup"}) + } + if err := AtomicWrite(dest, encoded); err != nil { + return err + } + kind := "inventory" + if dest != s.Output { + kind = "partial-inventory" + } + report.Artifacts = append(report.Artifacts, inventory.Artifact{Path: dest, Kind: kind}) + } + return s.Report(report) +} +func (s *Store) Report(report *inventory.Report) error { + p := filepath.Join(s.RunDir, "report.json") + b, err := json.MarshalIndent(report, "", " ") + if err != nil { + return err + } + if err := AtomicWrite(p, append(b, '\n')); err != nil { + return err + } + report.Artifacts = append(report.Artifacts, inventory.Artifact{Path: p, Kind: "report"}) + return nil +} + +// AtomicWrite uses restrictive same-directory temp files. Locks protect cooperating writers only. +func AtomicWrite(filename string, data []byte) error { + if err := safePath(filename); err != nil { + return err + } + f, err := os.CreateTemp(filepath.Dir(filename), ".l8k-dpf-*") + if err != nil { + return err + } + name := f.Name() + defer os.Remove(name) + if err = f.Chmod(0600); err == nil { + _, err = f.Write(data) + } + if err == nil { + err = f.Sync() + } + err = errors.Join(err, f.Close()) + if err != nil { + return err + } + if runtime.GOOS == "windows" { + return replaceFile(name, filename) + } + if err := os.Rename(name, filename); err != nil { + return err + } + directory, err := os.Open(filepath.Dir(filename)) + if err != nil { + return err + } + defer directory.Close() + return directory.Sync() +} diff --git a/pkg/dpf/artifacts/store_test.go b/pkg/dpf/artifacts/store_test.go new file mode 100644 index 00000000..9bcd993a --- /dev/null +++ b/pkg/dpf/artifacts/store_test.go @@ -0,0 +1,234 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package artifacts + +import ( + "context" + "encoding/json" + "fmt" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestArtifacts(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "DPF artifacts") } + +var _ = Describe("safe publication", func() { + var dir string + var input config.Input + BeforeEach(func() { + var e error + dir, e = os.MkdirTemp("", "dpf-artifacts-") + Expect(e).NotTo(HaveOccurred()) + dir, e = filepath.EvalSymlinks(dir) + Expect(e).NotTo(HaveOccurred()) + DeferCleanup(os.RemoveAll, dir) + source := []byte("# site intent\nmanagement:\n kubeconfigRef: {file: credentials/mgmt}\ncredentials: {}\ndiscovery:\n dpus: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: r}]\n hosts: [{bmcEndpoint: 'https://192.0.2.2', credentialRef: r}]\n") + p := filepath.Join(dir, "input.yaml") + Expect(os.WriteFile(p, source, 0600)).To(Succeed()) + input, e = config.Load(p) + Expect(e).NotTo(HaveOccurred()) + }) + report := func(complete bool) *inventory.Report { + outcome := "partial" + if complete { + outcome = "complete" + } + return &inventory.Report{Inventory: &inventory.Inventory{FormatVersion: "v1alpha1"}, Discovery: inventory.Discovery{Outcome: outcome}, Artifacts: []inventory.Artifact{}} + } + It("preserves normal handoff on partial collection and rebases references", func() { + s, e := New(input, "", false) + Expect(e).NotTo(HaveOccurred()) + r := report(false) + Expect(s.Publish(context.Background(), r)).To(Succeed()) + _, e = os.Stat(s.Output) + Expect(os.IsNotExist(e)).To(BeTrue()) + b, e := os.ReadFile(filepath.Join(s.RunDir, "partial.yaml")) + Expect(e).NotTo(HaveOccurred()) + Expect(string(b)).To(ContainSubstring("# site intent")) + Expect(string(b)).To(ContainSubstring("../../credentials/mgmt")) + st, e := os.Stat(filepath.Join(s.RunDir, "report.json")) + Expect(e).NotTo(HaveOccurred()) + Expect(st.Mode().Perm()).To(Equal(os.FileMode(0600))) + }) + It("omits BMC evidence bookkeeping from complete and partial saved inventories and reports", func() { + for _, complete := range []bool{true, false} { + output := filepath.Join(dir, fmt.Sprintf("no-evidence-%t.yaml", complete)) + store, err := New(input, output, true) + Expect(err).NotTo(HaveOccurred()) + r := report(complete) + r.Inventory.Hosts = []inventory.Device{{ID: "host-one", ObservationState: "observed", EvidenceRefs: []string{}, Adapters: []inventory.Adapter{{Key: "card-one", SerialNumber: "CARD-ONE", EvidenceRefs: []string{}}}, Firmware: []inventory.Firmware{{Component: "bmc", State: "known", RawVersion: "iLO 6 v1.50", Activation: "unknown", EvidenceRefs: []string{}}}}} + r.Inventory.DPUs = []inventory.Device{{ID: "dpu-one", SerialNumber: "CARD-ONE", ObservationState: "observed", Observations: map[string]inventory.Observation{"bmc-administrator": {State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED"}}, EvidenceRefs: []string{}}} + r.Inventory.Associations = []inventory.Association{{HostID: "host-one", DPUID: "dpu-one", HostAdapterKey: "card-one", EvidenceRefs: []string{}}} + r.Inventory.Evidence = map[string]inventory.Evidence{} + r.Discovery.Checks = []inventory.Check{{ID: "dpf.management.bmc-administrator", State: "passed", RequiredPhase: "discover", ReasonCode: "BMC_ADMIN_VERIFIED", EvidenceRefs: []string{}}} + Expect(store.Publish(context.Background(), r)).To(Succeed()) + path := output + if !complete { + path = filepath.Join(store.RunDir, "partial.yaml") + } + data, err := os.ReadFile(path) + Expect(err).NotTo(HaveOccurred()) + var saved map[string]any + Expect(yaml.Unmarshal(data, &saved)).To(Succeed()) + Expect(hasEvidenceKey(saved)).To(BeFalse()) + loaded, err := config.Load(path) + Expect(err).NotTo(HaveOccurred()) + Expect(loaded.Site.Inventory.Hosts[0].Firmware[0].RawVersion).To(Equal("iLO 6 v1.50")) + Expect(loaded.Site.Inventory.DPUs[0].Observations["bmc-administrator"].ReasonCode).To(Equal("BMC_ADMIN_VERIFIED")) + data, err = os.ReadFile(filepath.Join(store.RunDir, "report.json")) + Expect(err).NotTo(HaveOccurred()) + var detailed map[string]any + Expect(json.Unmarshal(data, &detailed)).To(Succeed()) + Expect(hasEvidenceKey(detailed)).To(BeFalse()) + } + }) + It("refreshes legacy BMC evidence without carrying it forward or changing the backup", func() { + // Construct the historical wire bytes directly: the current writer must + // not emit the obsolete bookkeeping even for a legacy-format object. + data := append(append([]byte(nil), input.Source...), []byte("\ninventory:\n formatVersion: v1alpha1\n evidence:\n legacy-ref: {kind: redfish, authenticated: true}\n hosts:\n - evidenceRefs: [legacy-ref]\n")...) + Expect(os.WriteFile(input.Path, data, 0600)).To(Succeed()) + previous, err := config.Load(input.Path) + Expect(err).NotTo(HaveOccurred()) + Expect(previous.Site.Inventory.Evidence).To(HaveKey("legacy-ref")) + store, err := New(previous, previous.Path, true) + Expect(err).NotTo(HaveOccurred()) + Expect(store.Publish(context.Background(), report(true))).To(Succeed()) + fresh, err := os.ReadFile(previous.Path) + Expect(err).NotTo(HaveOccurred()) + Expect(string(fresh)).NotTo(ContainSubstring("legacy-ref")) + backup, err := os.ReadFile(filepath.Join(store.RunDir, "source-before.yaml")) + Expect(err).NotTo(HaveOccurred()) + Expect(backup).To(Equal(data)) + }) + It("omits SSH evidence from config while preserving the diagnostic report", func() { + inv := &inventory.Inventory{Evidence: map[string]inventory.Evidence{"ssh-ref": {Kind: "ssh", Authenticated: true}}, Hosts: []inventory.Device{{SSHEndpoint: "ssh://192.0.2.10:22", EvidenceRefs: []string{"ssh-ref"}}}} + data, err := config.Saved(input, inv, filepath.Join(dir, "mixed.yaml")) + Expect(err).NotTo(HaveOccurred()) + var saved map[string]any + Expect(yaml.Unmarshal(data, &saved)).To(Succeed()) + Expect(hasEvidenceKey(saved)).To(BeFalse()) + Expect(string(data)).To(ContainSubstring("ssh://192.0.2.10:22")) + encoded, err := json.Marshal(inv) + Expect(err).NotTo(HaveOccurred()) + Expect(string(encoded)).To(ContainSubstring(`"evidenceRefs":["ssh-ref"]`)) + }) + + It("creates an in-place source backup before replacement", func() { + s, e := New(input, input.Path, true) + Expect(e).NotTo(HaveOccurred()) + Expect(s.Publish(context.Background(), report(true))).To(Succeed()) + b, e := os.ReadFile(filepath.Join(s.RunDir, "source-before.yaml")) + Expect(e).NotTo(HaveOccurred()) + Expect(b).To(Equal(input.Source)) + }) + It("rejects concurrent source edits before replacing output", func() { + s, e := New(input, "", false) + Expect(e).NotTo(HaveOccurred()) + Expect(os.WriteFile(input.Path, []byte("changed"), 0600)).To(Succeed()) + Expect(s.Publish(context.Background(), report(true))).NotTo(Succeed()) + _, e = os.Stat(s.Output) + Expect(os.IsNotExist(e)).To(BeTrue()) + }) + It("rejects existing unrelated output and symlinked run directories", func() { + out := filepath.Join(dir, "out.yaml") + Expect(os.WriteFile(out, []byte("unrelated: true"), 0600)).To(Succeed()) + _, e := New(input, out, true) + Expect(e).To(HaveOccurred()) + Expect(os.Remove(out)).To(Succeed()) + Expect(os.Symlink(dir, filepath.Join(dir, "out.discovery"))).To(Succeed()) + _, e = New(input, out, true) + Expect(e).To(HaveOccurred()) + }) + It("keeps a stable lock and detects a second cooperating writer", func() { + a, e := New(input, "", false) + Expect(e).NotTo(HaveOccurred()) + b, e := New(input, "", false) + Expect(e).NotTo(HaveOccurred()) + Expect(a.Publish(context.Background(), report(true))).To(Succeed()) + Expect(b.Publish(context.Background(), report(true))).NotTo(Succeed()) + _, e = os.Stat(a.Output + ".lock") + Expect(e).NotTo(HaveOccurred()) + }) + It("checks the destination baseline even when source already contains inventory", func() { + output := filepath.Join(dir, "prior.yaml") + prior := &inventory.Inventory{FormatVersion: "v1alpha1", ScopeDigest: "cluster-B"} + saved, e := config.Saved(input, prior, output) + Expect(e).NotTo(HaveOccurred()) + Expect(os.WriteFile(output, saved, 0600)).To(Succeed()) + input.Site.Inventory = &inventory.Inventory{FormatVersion: "v1alpha1", ScopeDigest: "cluster-A"} + store, e := New(input, output, true) + Expect(e).NotTo(HaveOccurred()) + r := report(true) + r.Inventory.ScopeDigest = "cluster-A" + Expect(store.Publish(context.Background(), r)).NotTo(Succeed()) + unchanged, e := os.ReadFile(output) + Expect(e).NotTo(HaveOccurred()) + Expect(unchanged).To(Equal(saved)) + }) + It("does not overwrite a destination collected with changed or unrecorded Redfish trust", func() { + for index, priorTrust := range []string{"", "system-roots", "custom-ca-before", "verification-disabled"} { + output := filepath.Join(dir, []string{"legacy.yaml", "system.yaml", "custom.yaml", "insecure.yaml"}[index]) + prior := &inventory.Inventory{FormatVersion: "v1alpha1", Collection: inventory.Collection{Mode: "inventory"}, RequestedScope: []inventory.ScopeEndpoint{{Role: "hosts", Endpoint: "https://192.0.2.2", TLSTrustDigest: priorTrust}}} + discovery.Canonicalize(prior) + saved, err := config.Saved(input, prior, output) + Expect(err).NotTo(HaveOccurred()) + Expect(os.WriteFile(output, saved, 0600)).To(Succeed()) + store, err := New(input, output, true) + Expect(err).NotTo(HaveOccurred()) + r := report(true) + r.Inventory.Collection.Mode = "inventory" + r.Inventory.RequestedScope = []inventory.ScopeEndpoint{{Role: "hosts", Endpoint: "https://192.0.2.2", TLSTrustDigest: "custom-ca-after"}} + discovery.Canonicalize(r.Inventory) + Expect(store.Publish(context.Background(), r)).To(MatchError(ContainSubstring("conflicts with resolved scope"))) + unchanged, err := os.ReadFile(output) + Expect(err).NotTo(HaveOccurred()) + Expect(unchanged).To(Equal(saved)) + } + }) + It("preserves a legacy destination even when a fresh source would otherwise permit publication", func() { + output := filepath.Join(dir, "legacy.yaml") + key := inventory.Digest("dpf-adapter-v1", []string{"host", "CARD-ONE"}) + prior := &inventory.Inventory{ + FormatVersion: "v1alpha1", + Hosts: []inventory.Device{{ID: "host", Adapters: []inventory.Adapter{{Key: key, SerialNumber: "CARD-ONE"}}}}, + DPUs: []inventory.Device{{ID: "card", ReaderAdapter: "host-card-v1", RecordKey: inventory.Digest("dpf-host-card-v1", []string{"host", key})}}, + } + saved, err := config.Saved(input, prior, output) + Expect(err).NotTo(HaveOccurred()) + Expect(os.WriteFile(output, saved, 0600)).To(Succeed()) + store, err := New(input, output, true) + Expect(err).NotTo(HaveOccurred()) + Expect(store.Publish(context.Background(), report(true))).To(MatchError(ContainSubstring("legacy Redfish card bindings"))) + unchanged, err := os.ReadFile(output) + Expect(err).NotTo(HaveOccurred()) + Expect(unchanged).To(Equal(saved)) + }) + +}) + +// Inspect parsed documents so scalar diagnostic text never resembles a field match. +func hasEvidenceKey(value any) bool { + switch v := value.(type) { + case map[string]any: + for key, child := range v { + if strings.EqualFold(key, "evidence") || strings.EqualFold(key, "evidenceRef") || strings.EqualFold(key, "evidenceRefs") || hasEvidenceKey(child) { + return true + } + } + case []any: + for _, child := range v { + if hasEvidenceKey(child) { + return true + } + } + } + return false +} diff --git a/pkg/dpf/catalog/catalog.go b/pkg/dpf/catalog/catalog.go new file mode 100644 index 00000000..3da825a9 --- /dev/null +++ b/pkg/dpf/catalog/catalog.go @@ -0,0 +1,29 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package catalog owns qualified discovery assets. Site inputs cannot waive these gates. +package catalog + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/probe" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" +) + +// ProbeAssets pins the existing netshoot v0.16 image. Registry and local non-root +// tool checks do not qualify Kubernetes admission, so activation remains gated. +// No discovery image is built or published by Launch Kit. +func ProbeAssets() probe.Assets { + return probe.Assets{ + Repository: "docker.io/nicolaka/netshoot", + ManifestDigest: "sha256:b09d9b21381f47a79b3cbcb30da25266dc17186ea00ae65e99fdc51396f48e70", + PlatformDigests: map[string]string{ + "amd64": "sha256:147f2c4cb0aa7a741151bce2bfef71d9e1a04631636e24665ee2a5ed799feb47", + "arm64": "sha256:f460dc5e85890cad42b14973f5724b40831cbb3e1f745470caf6b8cf0d6ec7fe", + }, + Qualified: false, + } +} + +// Hardware is empty until real resource graphs and adapter completeness predicates +// have been qualified for specific models/firmware. Candidate reads remain useful. +func Hardware() []redfish.Qualification { return nil } diff --git a/pkg/dpf/config/compact_inventory_test.go b/pkg/dpf/config/compact_inventory_test.go new file mode 100644 index 00000000..3bc9ab1d --- /dev/null +++ b/pkg/dpf/config/compact_inventory_test.go @@ -0,0 +1,52 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package config + +import ( + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("compact discovery inventory input", func() { + It("restores compact evidence after strict YAML decoding", func() { + in, err := Decode([]byte(seed), "/site/input.yaml") + Expect(err).NotTo(HaveOccurred()) + inv := &inventory.Inventory{FormatVersion: inventory.FormatVersion, DPUs: []inventory.Device{{ + ID: "dpu-one", BMCEndpoint: "https://dpu.invalid", Model: "B4240", ReaderAdapter: "bluefield-card-v1", ObservationState: "observed", Observations: map[string]inventory.Observation{ + "bmc-administrator": {State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED"}, + "firmware-inventory": {State: "observed"}, + "psid": {State: "observed", Values: []string{"MT_0000001774"}, ReasonCode: "PSID_OBSERVED", Tool: "redfish"}, + }, + }}} + encoded, err := Saved(in, inv, in.Path) + Expect(err).NotTo(HaveOccurred()) + Expect(string(encoded)).NotTo(ContainSubstring("observationState: observed")) + restored, err := Decode(encoded, in.Path) + Expect(err).NotTo(HaveOccurred()) + d := restored.Site.Inventory.DPUs[0] + Expect(d.ObservationState).To(Equal("observed")) + Expect(d.Observations["bmc-administrator"].ReasonCode).To(Equal("BMC_ADMIN_VERIFIED")) + Expect(d.Observations["firmware-inventory"].State).To(Equal("observed")) + Expect(d.Observations["psid"]).To(Equal(inv.DPUs[0].Observations["psid"])) + }) + + It("continues rejecting unknown fields inside compact device and authentication records", func() { + for _, field := range []string{" deviceTypo: value\n", " firmware: {firmwareTypo: value}\n", " runtime: {runtimeTypo: value}\n"} { + _, err := Decode([]byte(seed+"inventory:\n formatVersion: v1alpha3\n dpus:\n - id: dpu-one\n"+field), "/site/input.yaml") + Expect(err).To(HaveOccurred()) + } + }) + + It("does not infer success from omitted legacy state", func() { + text := seed + "inventory:\n formatVersion: v1alpha1\n dpus: [{id: dpu-one}]\n" + in, err := Decode([]byte(text), "/site/input.yaml") + Expect(err).NotTo(HaveOccurred()) + Expect(in.Site.Inventory.DPUs[0].ObservationState).To(BeEmpty()) + _, err = Decode([]byte(strings.Replace(text, "id: dpu-one", "id: dpu-one, typo: value", 1)), "/site/input.yaml") + Expect(err).To(HaveOccurred()) + }) +}) diff --git a/pkg/dpf/config/config.go b/pkg/dpf/config/config.go new file mode 100644 index 00000000..cb9970a3 --- /dev/null +++ b/pkg/dpf/config/config.go @@ -0,0 +1,644 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package config loads ordinary DPF site YAML without Host defaults or ambient access. +package config + +import ( + "bytes" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "io" + "k8s.io/apimachinery/pkg/util/validation" + "net" + "net/url" + "os" + "path" + "path/filepath" + "strconv" + "strings" + "time" + "unicode/utf8" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/addresslookup" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" + "github.com/nvidia/k8s-launch-kit/pkg/yamlinput" + yaml "gopkg.in/yaml.v3" +) + +type FileRef struct { + File string `yaml:"file"` +} +type PasswordRef struct { + Env *string `yaml:"env,omitempty"` + File *string `yaml:"file,omitempty"` +} +type Credential struct { + Username string `yaml:"username"` + PasswordRef PasswordRef `yaml:"passwordRef,omitempty"` + PrivateKeyRef *FileRef `yaml:"privateKeyRef,omitempty"` +} +type TLS struct { + CAFile *string `yaml:"caFile,omitempty"` + InsecureSkipVerify *bool `yaml:"insecureSkipVerify,omitempty"` +} +type SSH struct { + Address string `yaml:"address"` + KnownHostsFile string `yaml:"knownHostsFile"` + Sudo *bool `yaml:"sudo,omitempty"` +} +type Endpoint struct { + SSH *SSH `yaml:"ssh,omitempty"` + Name *string `yaml:"name,omitempty"` + BMCEndpoint string `yaml:"bmcEndpoint"` + CredentialRef string `yaml:"credentialRef"` + SystemURI *string `yaml:"systemURI,omitempty"` + TLS *TLS `yaml:"tls,omitempty"` +} +type PullSecret struct { + Name string `yaml:"name"` +} +type Probes struct { + Mode *string `yaml:"mode,omitempty"` + Namespace *string `yaml:"namespace,omitempty"` + ImageRepository *string `yaml:"imageRepository,omitempty"` + ImagePullSecrets []PullSecret `yaml:"imagePullSecrets,omitempty"` +} +type Limits struct { + Timeout *string `yaml:"timeout,omitempty"` + RequestTimeout *string `yaml:"requestTimeout,omitempty"` + MaxConcurrentBMCs *int `yaml:"maxConcurrentBMCs,omitempty"` +} +type Site struct { + DPF struct { + Release *string `yaml:"release,omitempty"` + } `yaml:"dpf,omitempty"` + Management struct { + KubeconfigRef FileRef `yaml:"kubeconfigRef"` + Context *string `yaml:"context,omitempty"` + } `yaml:"management"` + Credentials map[string]Credential `yaml:"credentials"` + Discovery struct { + AddressLookup *addresslookup.Config `yaml:"addressLookup,omitempty"` + DPUs []Endpoint `yaml:"dpus"` + Hosts []Endpoint `yaml:"hosts"` + TLS *TLS `yaml:"tls,omitempty"` + Probes Probes `yaml:"probes,omitempty"` + Limits Limits `yaml:"limits,omitempty"` + } `yaml:"discovery"` + Inventory *inventory.Inventory `yaml:"inventory,omitempty"` +} +type Input struct { + Site Site + Source []byte + Path string + Document *yaml.Node +} + +// Access is private runtime state. It deliberately cannot be serialized or logged. +type Access struct { + username string + password string + tls *tls.Config + ca []byte +} + +func (a Access) Credentials() (string, string) { return a.username, a.password } +func (a Access) TLSConfig() *tls.Config { + if a.tls == nil { + return nil + } + config := a.tls.Clone() + if config.RootCAs != nil { + config.RootCAs = config.RootCAs.Clone() + } + return config +} +func (a Access) PublicCA() []byte { return bytes.Clone(a.ca) } +func (a Access) String() string { return "" } +func (a Access) InsecureSkipVerify() bool { return a.tls != nil && a.tls.InsecureSkipVerify } + +// TrustDigest identifies the resolved verification policy, without file paths or +// CA bytes. Custom roots augment system trust; the external system trust store +// itself is not fingerprinted. The CA snapshot cannot change after resolution. +func (a Access) TrustDigest() string { + if a.InsecureSkipVerify() { + return "verification-disabled" + } + if len(a.ca) == 0 { + return "system-roots" + } + return inventory.Digest("dpf-redfish-trust-v1", []string{"system-roots+custom-ca", string(a.ca)}) +} + +type Target struct { + Name string + SSH *SSHAccess + Endpoint string + SystemURI string + InputRef string + Role string + Access Access +} + +// DisplayName is presentation only; names and URL-private components never +// participate in physical identity or access-policy scope. +func (t Target) DisplayName() string { + endpoint := "" + if u, err := url.Parse(t.Endpoint); err == nil && (u.Scheme == "https" || u.Scheme == "ssh") && u.Hostname() != "" { + endpoint = (&url.URL{Scheme: u.Scheme, Host: u.Host}).String() + } + if t.Name != "" { + return strconv.Quote(t.Name) + " (" + endpoint + ")" + } + return endpoint +} + +type Request struct { + Software *releases.Release + AddressLookup *addresslookup.Request + Input Input + Targets []Target + Kubeconfig string + Context string + Mode string + Namespace string + ImageRepository string + ImagePullSecrets []string + Timeout time.Duration + RequestTimeout time.Duration + MaxConcurrentBMCs int +} + +func regularFile(p string, limit int64) ([]byte, error) { + before, statErr := os.Stat(p) + if statErr != nil || !before.Mode().IsRegular() || before.Size() > limit { + return nil, fmt.Errorf("reference must be a bounded regular file") + } + f, err := os.Open(p) + if err != nil { + return nil, fmt.Errorf("cannot open referenced file") + } + defer f.Close() + st, err := f.Stat() + if err != nil || !st.Mode().IsRegular() || st.Size() > limit { + return nil, fmt.Errorf("reference must be a bounded regular file") + } + b, err := io.ReadAll(io.LimitReader(f, limit+1)) + if err != nil || int64(len(b)) > limit { + return nil, fmt.Errorf("cannot read bounded file") + } + return b, nil +} +func Load(p string) (Input, error) { + abs, err := filepath.Abs(p) + if err != nil || p == "" || p == "-" { + return Input{}, fmt.Errorf("--user-config must name one local YAML file") + } + b, err := regularFile(abs, 16<<20) + if err != nil { + return Input{}, err + } + in, err := Decode(b, abs) + if err != nil { + return Input{}, fmt.Errorf("%s: %w", abs, err) + } + return in, nil +} +func Decode(b []byte, p string) (Input, error) { + if len(b) > 16<<20 { + return Input{}, fmt.Errorf("YAML input exceeds 16 MiB") + } + dec := yaml.NewDecoder(bytes.NewReader(b)) + var doc yaml.Node + if err := dec.Decode(&doc); err != nil { + return Input{}, fmt.Errorf("invalid YAML") + } + var extra yaml.Node + if err := dec.Decode(&extra); !errors.Is(err, io.EOF) { + return Input{}, fmt.Errorf("expected one YAML document") + } + if len(doc.Content) != 1 || doc.Content[0].Kind != yaml.MappingNode { + return Input{}, fmt.Errorf("expected mapping document") + } + if err := validateKeys(&doc, map[*yaml.Node]bool{}, map[*yaml.Node]bool{}, 0); err != nil { + return Input{}, err + } + expanded, err := yamlinput.Expand(&doc) + if err != nil { + return Input{}, err + } + // Scalar types belong to their effective paths after merge/alias expansion. + // Expand preserves YAML tags, so validation still rejects string coercion. + if err := validateScalarTypes(expanded.Content[0], ""); err != nil { + return Input{}, err + } + encoded, err := yamlinput.Encode(expanded) + if err != nil { + return Input{}, err + } + var site Site + strict := yaml.NewDecoder(bytes.NewReader(encoded)) + strict.KnownFields(true) + if err := strict.Decode(&site); err != nil { + return Input{}, fmt.Errorf("unknown field or invalid type in DPF configuration") + } + if err := inventory.NormalizeInventory(site.Inventory); err != nil { + return Input{}, err + } + return Input{Site: site, Source: bytes.Clone(b), Path: p, Document: expanded}, nil +} +func validateKeys(n *yaml.Node, active, visited map[*yaml.Node]bool, depth int) error { + if n == nil || depth > 64 || active[n] { + return fmt.Errorf("cyclic or excessive YAML aliases") + } + if visited[n] { + return nil + } + visited[n] = true + active[n] = true + defer delete(active, n) + if n.Kind == yaml.AliasNode { + return validateKeys(n.Alias, active, visited, depth+1) + } + if n.Kind == yaml.MappingNode { + seen := map[string]bool{} + for i := 0; i < len(n.Content); i += 2 { + k := n.Content[i] + if (k.Tag != "!!str" && k.Tag != "!!merge") || seen[k.Value] { + return fmt.Errorf("duplicate or non-string key at line %d", k.Line) + } + seen[k.Value] = true + } + } + for _, c := range n.Content { + if err := validateKeys(c, active, visited, depth+1); err != nil { + return err + } + } + return nil +} +func Resolve(in Input) (Request, error) { + r := Request{Input: in, Timeout: 10 * time.Minute, RequestTimeout: 15 * time.Second, MaxConcurrentBMCs: 4, Mode: "management"} + s := in.Site + base := filepath.Dir(in.Path) + release := "" + if s.DPF.Release != nil { + if *s.DPF.Release == "" { + return r, fmt.Errorf("/dpf/release must not be empty") + } + release = *s.DPF.Release + } + software, err := releases.Resolve(release) + if err != nil { + return r, err + } + r.Software = &software + if s.Discovery.AddressLookup != nil { + lookup, err := addresslookup.Validate(*s.Discovery.AddressLookup) + if err != nil { + return r, err + } + for i := range lookup.Sources { + lookup.Sources[i].File = resolvePath(base, lookup.Sources[i].File) + } + r.AddressLookup = &lookup + } + if s.Management.KubeconfigRef.File != "" { + r.Kubeconfig = resolvePath(base, s.Management.KubeconfigRef.File) + } + if s.Management.Context != nil { + if *s.Management.Context == "" { + return r, fmt.Errorf("/management/context must not be empty") + } + r.Context = *s.Management.Context + } + p := s.Discovery.Probes + if p.Mode != nil { + r.Mode = *p.Mode + } + if r.Mode != "management" && r.Mode != "inventory" { + return r, fmt.Errorf("/discovery/probes/mode must be management or inventory") + } + if r.Mode == "management" && r.Kubeconfig == "" { + return r, fmt.Errorf("/management/kubeconfigRef/file is required in management mode") + } + if r.Kubeconfig == "" && s.Management.Context != nil { + return r, fmt.Errorf("/management/context requires kubeconfigRef") + } + if p.Namespace != nil { + r.Namespace = *p.Namespace + if r.Namespace == "" || len(r.Namespace) > 63 || !dnsLabel(r.Namespace) { + return r, fmt.Errorf("invalid /discovery/probes/namespace") + } + } + if p.ImageRepository != nil { + r.ImageRepository = *p.ImageRepository + if r.ImageRepository == "" || strings.ContainsAny(r.ImageRepository, "@ \t\n") || strings.Contains(r.ImageRepository, "://") { + return r, fmt.Errorf("invalid probe repository") + } + last := r.ImageRepository[strings.LastIndex(r.ImageRepository, "/")+1:] + if strings.Contains(last, ":") { + return r, fmt.Errorf("probe repository must not have a tag") + } + } + for _, secret := range p.ImagePullSecrets { + if p.Namespace == nil || secret.Name == "" || !dnsLabel(secret.Name) { + return r, fmt.Errorf("imagePullSecrets requires an explicit namespace and valid names") + } + r.ImagePullSecrets = append(r.ImagePullSecrets, secret.Name) + } + l := s.Discovery.Limits + for _, f := range []struct { + value *string + dest *time.Duration + max time.Duration + }{{l.Timeout, &r.Timeout, time.Hour}, {l.RequestTimeout, &r.RequestTimeout, 2 * time.Minute}} { + if f.value != nil { + d, err := time.ParseDuration(*f.value) + if err != nil || d < time.Second || d > f.max { + return r, fmt.Errorf("invalid /discovery/limits duration") + } + *f.dest = d + } + } + if l.MaxConcurrentBMCs != nil { + r.MaxConcurrentBMCs = *l.MaxConcurrentBMCs + } + if r.RequestTimeout > r.Timeout || r.MaxConcurrentBMCs < 1 || r.MaxConcurrentBMCs > 16 { + return r, fmt.Errorf("invalid /discovery/limits") + } + if len(s.Discovery.Hosts) == 0 || (r.Mode == "management" && len(s.Discovery.DPUs) == 0) || len(s.Discovery.DPUs)+len(s.Discovery.Hosts) > 256 { + return r, fmt.Errorf("discovery requires explicit hosts, at most 256 endpoints, and explicit DPUs in management mode") + } + seen := map[string]bool{} + type accessKey struct { + credential, caPath string + insecure bool + } + accesses := map[accessKey]Access{} + for _, group := range []struct { + role string + endpoints []Endpoint + }{{"dpus", s.Discovery.DPUs}, {"hosts", s.Discovery.Hosts}} { + for i, e := range group.endpoints { + ptr := fmt.Sprintf("/discovery/%s/%d", group.role, i) + name := "" + if e.Name != nil { + name = *e.Name + } + if e.SSH != nil { + if r.Mode != "inventory" { + return r, fmt.Errorf("%s/ssh requires inventory mode; management probes require BMC targets", ptr) + } + if e.BMCEndpoint != "" || e.SystemURI != nil || e.TLS != nil { + return r, fmt.Errorf("%s must select exactly one SSH or Redfish source", ptr) + } + origin, err := SSHOrigin(e.SSH.Address) + if err != nil { + return r, fmt.Errorf("%s/ssh/address: %w", ptr, err) + } + if seen[origin] { + return r, fmt.Errorf("duplicate SSH endpoint at %s", ptr) + } + seen[origin] = true + c, ok := s.Credentials[e.CredentialRef] + if !ok { + return r, fmt.Errorf("%s/credentialRef is unknown", ptr) + } + access, err := resolveSSHAccess(c, *e.SSH, base) + if err != nil { + return r, fmt.Errorf("%s/ssh: %w", ptr, err) + } + r.Targets = append(r.Targets, Target{Name: name, Endpoint: origin, InputRef: ptr, Role: group.role, SSH: access}) + continue + } + origin, err := Origin(e.BMCEndpoint) + if err != nil { + return r, fmt.Errorf("%s/bmcEndpoint: %w", ptr, err) + } + if seen[origin] { + return r, fmt.Errorf("duplicate BMC origin at %s", ptr) + } + seen[origin] = true + selector := "" + if e.SystemURI != nil { + selector = *e.SystemURI + if err := ResourceURI(selector); err != nil { + return r, fmt.Errorf("%s/systemURI: invalid Redfish resource", ptr) + } + } + c, ok := s.Credentials[e.CredentialRef] + if !ok || c.Username == "" { + return r, fmt.Errorf("%s/credentialRef must reference a username/passwordRef", ptr) + } + policy := s.Discovery.TLS + if e.TLS != nil { + policy = e.TLS + } + caPath := "" + if policy != nil && policy.CAFile != nil { + if *policy.CAFile == "" { + return r, fmt.Errorf("empty caFile") + } + caPath = resolvePath(base, *policy.CAFile) + } + insecure := policy != nil && policy.InsecureSkipVerify != nil && *policy.InsecureSkipVerify + if insecure && caPath != "" { + return r, fmt.Errorf("%s/tls: caFile and insecureSkipVerify: true are mutually exclusive", ptr) + } + cacheKey := accessKey{credential: e.CredentialRef, caPath: caPath, insecure: insecure} + a, ok := accesses[cacheKey] + if !ok { + a, err = resolveAccess(c, caPath, base, insecure) + if err != nil { + return r, fmt.Errorf("%s: %w", ptr, err) + } + accesses[cacheKey] = a + } + r.Targets = append(r.Targets, Target{Name: name, Endpoint: origin, SystemURI: selector, InputRef: ptr, Role: group.role, Access: a}) + } + } + return r, nil +} +func dnsLabel(s string) bool { return len(validation.IsDNS1123Label(s)) == 0 } +func resolvePath(base, p string) string { + if filepath.IsAbs(p) { + return filepath.Clean(p) + } + return filepath.Join(base, p) +} +func resolveAccess(c Credential, caPath, base string, insecure bool) (Access, error) { + if c.PrivateKeyRef != nil { + return Access{}, fmt.Errorf("redfish requires passwordRef, not privateKeyRef") + } + password, err := resolvePassword(c.PasswordRef, base) + if err != nil { + return Access{}, err + } + roots, err := x509.SystemCertPool() + if err != nil || roots == nil { + roots = x509.NewCertPool() + } + var ca []byte + if caPath != "" { + ca, err = regularFile(caPath, 1<<20) + if err != nil || !roots.AppendCertsFromPEM(ca) { + return Access{}, fmt.Errorf("invalid CA file") + } + } + return Access{username: c.Username, password: password, tls: &tls.Config{MinVersion: tls.VersionTLS12, RootCAs: roots, InsecureSkipVerify: insecure}, ca: ca}, nil +} +func resolvePassword(p PasswordRef, base string) (string, error) { + if (p.Env == nil) == (p.File == nil) { + return "", fmt.Errorf("passwordRef requires exactly one env or file") + } + var password string + if p.Env != nil { + if *p.Env == "" { + return "", fmt.Errorf("empty passwordRef.env") + } + password = os.Getenv(*p.Env) + } else { + if *p.File == "" { + return "", fmt.Errorf("empty passwordRef.file") + } + b, err := regularFile(resolvePath(base, *p.File), 64<<10) + if err != nil { + return "", fmt.Errorf("cannot resolve password file") + } + if !utf8.Valid(b) { + return "", fmt.Errorf("password file must be UTF-8") + } + password = string(b) + if strings.HasSuffix(password, "\n") { + password = strings.TrimSuffix(strings.TrimSuffix(password, "\n"), "\r") + } + } + if password == "" || len(password) > 64<<10 { + return "", fmt.Errorf("password reference is missing, empty or too large") + } + return password, nil +} +func Origin(s string) (string, error) { + u, err := url.Parse(s) + if err != nil || u.Scheme != "https" || u.Hostname() == "" || u.User != nil || u.RawQuery != "" || u.Fragment != "" || u.Opaque != "" || (u.Path != "" && u.Path != "/") { + return "", fmt.Errorf("expected HTTPS origin without userinfo/path/query/fragment") + } + host := strings.ToLower(u.Hostname()) + port := u.Port() + if port != "" { + n, e := strconv.Atoi(port) + if e != nil || n < 1 || n > 65535 { + return "", fmt.Errorf("invalid port") + } + } + if port == "" || port == "443" { + if strings.Contains(host, ":") { + host = "[" + host + "]" + } + } else { + host = net.JoinHostPort(host, port) + } + return "https://" + host, nil +} +func ResourceURI(s string) error { + u, e := url.Parse(s) + if e != nil || u.IsAbs() || u.Host != "" || !strings.HasPrefix(u.Path, "/redfish/v1/") || u.RawQuery != "" || u.Fragment != "" || strings.Contains(u.Path, "..") || path.Clean(u.Path) != u.Path || u.RawPath != "" { + return fmt.Errorf("invalid resource URI") + } + return nil +} + +// Saved patches only generated inventory and file references whose base changed. +func Saved(in Input, inv *inventory.Inventory, destination string) ([]byte, error) { + doc, err := yamlinput.Expand(in.Document) + if err != nil { + return nil, err + } + var replacement yaml.Node + if err := replacement.Encode(inv); err != nil { + return nil, err + } + yamlinput.Set(doc.Content[0], "inventory", &replacement) + if filepath.Dir(destination) != filepath.Dir(in.Path) { + if err := rebase(doc.Content[0], filepath.Dir(in.Path), filepath.Dir(destination), ""); err != nil { + return nil, err + } + } + return yamlinput.Encode(doc) +} +func rebase(n *yaml.Node, oldBase, newBase, p string) error { + if n.Kind == yaml.MappingNode { + for i := 0; i < len(n.Content); i += 2 { + key := n.Content[i].Value + value := n.Content[i+1] + next := p + "/" + key + if key == "inventory" && p == "" { + continue + } + isFile := key == "caFile" || key == "knownHostsFile" || key == "file" && (p == "/discovery/addressLookup/sources" || strings.HasSuffix(p, "kubeconfigRef") || strings.HasSuffix(p, "passwordRef") || strings.HasSuffix(p, "privateKeyRef")) + if isFile && value.Kind == yaml.ScalarNode && value.Tag == "!!str" && value.Value != "" { + abs := resolvePath(oldBase, value.Value) + rel, err := filepath.Rel(newBase, abs) + if err != nil { + return err + } + value.Value = rel + } else if err := rebase(value, oldBase, newBase, next); err != nil { + return err + } + } + } else if n.Kind == yaml.SequenceNode { + for _, v := range n.Content { + if err := rebase(v, oldBase, newBase, p); err != nil { + return err + } + } + } + return nil +} + +func validateScalarTypes(n *yaml.Node, p string) error { + if p == "/inventory" { + return nil + } + if n.Kind == yaml.AliasNode { + return nil + } + switch n.Kind { + case yaml.ScalarNode: + if n.Tag == "!!null" { + return nil + } + expected := "!!str" + if p == "/discovery/limits/maxConcurrentBMCs" { + expected = "!!int" + } + if strings.HasSuffix(p, "/tls/insecureSkipVerify") || strings.HasSuffix(p, "/ssh/sudo") { + expected = "!!bool" + } + if n.Tag != expected { + return fmt.Errorf("%s has invalid scalar type at line %d", p, n.Line) + } + case yaml.MappingNode: + for i := 0; i < len(n.Content); i += 2 { + if err := validateScalarTypes(n.Content[i+1], p+"/"+n.Content[i].Value); err != nil { + return err + } + } + case yaml.SequenceNode: + for i, v := range n.Content { + if err := validateScalarTypes(v, fmt.Sprintf("%s/%d", p, i)); err != nil { + return err + } + } + } + return nil +} + +// ReadReference rejects streams/devices and bounds local reference reads. +func ReadReference(filename string, limit int64) ([]byte, error) { return regularFile(filename, limit) } diff --git a/pkg/dpf/config/config_test.go b/pkg/dpf/config/config_test.go new file mode 100644 index 00000000..b1b9a8d8 --- /dev/null +++ b/pkg/dpf/config/config_test.go @@ -0,0 +1,227 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package config + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestConfig(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "DPF configuration") } + +const seed = `# retained comment +management: + kubeconfigRef: {file: credentials/mgmt} +credentials: + read: + username: reader + passwordRef: {env: DPF_TEST_PASSWORD} +discovery: + dpus: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: read}] + hosts: [{bmcEndpoint: 'https://192.0.2.2', credentialRef: read}] +` + +var _ = Describe("DPF input", func() { + BeforeEach(func() { + Expect(os.Setenv("DPF_TEST_PASSWORD", " canary \n")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_TEST_PASSWORD") + }) + It("retains endpoint names for presentation without changing access scope", func() { + in, err := Decode([]byte(seed), "/site/input.yaml") + Expect(err).NotTo(HaveOccurred()) + before, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + name := "Rack A DPU" + in.Site.Discovery.DPUs[0].Name = &name + after, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + Expect(after.Targets[0].Name).To(Equal(name)) + Expect(after.Targets[0].DisplayName()).To(Equal(`"Rack A DPU" (https://192.0.2.1)`)) + Expect(Scope(after.Targets[0])).To(Equal(Scope(before.Targets[0]))) + }) + It("elides private URL components and escapes control characters in display labels", func() { + target := Target{Name: "Rack\nA\x1b", Endpoint: "https://user-canary:password-canary@host.example:443/private-path-canary?token=query-canary#fragment-canary"} + Expect(target.DisplayName()).To(Equal(`"Rack\nA\x1b" (https://host.example:443)`)) + Expect(Target{Endpoint: "ssh://host.example:22"}.DisplayName()).To(Equal("ssh://host.example:22")) + Expect(Target{Endpoint: "invalid\nendpoint"}.DisplayName()).To(Equal("")) + }) + It("uses bounded defaults and keeps private values out of saved YAML", func() { + in, e := Decode([]byte(seed), "/site/input.yaml") + Expect(e).NotTo(HaveOccurred()) + r, e := Resolve(in) + Expect(e).NotTo(HaveOccurred()) + Expect(r.Timeout).To(Equal(10 * time.Minute)) + _, password := r.Targets[0].Access.Credentials() + Expect(password).To(Equal(" canary \n")) + b, e := Saved(in, &inventory.Inventory{FormatVersion: "v1alpha1"}, "/results/out.yaml") + Expect(e).NotTo(HaveOccurred()) + Expect(string(b)).To(ContainSubstring("# retained comment")) + Expect(string(b)).To(ContainSubstring("../site/credentials/mgmt")) + Expect(string(b)).NotTo(ContainSubstring("canary")) + }) + It("rejects duplicate, unknown, cyclic and multiple documents", func() { + for _, input := range []string{seed + "management: {}\n", strings.Replace(seed, "discovery:", "typo:", 1), "a: &x {b: *x}", seed + "---\nfoo: bar\n"} { + _, e := Decode([]byte(input), "input.yaml") + Expect(e).To(HaveOccurred()) + } + }) + It("does not replace explicit zero, empty mode or empty TLS path with defaults", func() { + for _, addition := range []string{" limits: {maxConcurrentBMCs: 0}\n", " probes: {mode: ''}\n", " tls: {caFile: ''}\n"} { + in, e := Decode([]byte(seed+addition), "/site/in.yaml") + Expect(e).NotTo(HaveOccurred()) + _, e = Resolve(in) + Expect(e).To(HaveOccurred()) + } + }) + It("uses null as omitted", func() { + in, e := Decode([]byte(seed+" limits: {maxConcurrentBMCs: null}\n"), "/site/in.yaml") + Expect(e).NotTo(HaveOccurred()) + r, e := Resolve(in) + Expect(e).NotTo(HaveOccurred()) + Expect(r.MaxConcurrentBMCs).To(Equal(4)) + }) + It("validates merged scalar values at their effective paths and preserves explicit overrides", func() { + for _, addition := range []string{ + " limits: {<<: {maxConcurrentBMCs: 2}}\n", + " limits: {<<: [{maxConcurrentBMCs: 2}, {maxConcurrentBMCs: 3}]}\n", + " limits: {<<: {maxConcurrentBMCs: wrong}, maxConcurrentBMCs: 2}\n", + } { + in, err := Decode([]byte(seed+addition+" tls: {<<: {insecureSkipVerify: true}, insecureSkipVerify: false}\n"), "/site/in.yaml") + Expect(err).NotTo(HaveOccurred()) + r, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + Expect(r.MaxConcurrentBMCs).To(Equal(2)) + Expect(r.Targets[0].Access.InsecureSkipVerify()).To(BeFalse()) + b, err := Saved(in, &inventory.Inventory{FormatVersion: "v1alpha1"}, "/site/out.yaml") + Expect(err).NotTo(HaveOccurred()) + again, err := Decode(b, "/site/out.yaml") + Expect(err).NotTo(HaveOccurred()) + resolved, err := Resolve(again) + Expect(err).NotTo(HaveOccurred()) + Expect(resolved.MaxConcurrentBMCs).To(Equal(2)) + Expect(resolved.Targets[0].Access.InsecureSkipVerify()).To(BeFalse()) + } + }) + It("rejects invalid effective merged types and duplicate keys before expansion", func() { + for _, addition := range []string{ + " limits: {<<: {maxConcurrentBMCs: '2'}}\n", + " tls: {<<: {insecureSkipVerify: 'true'}}\n", + " limits: {<<: {maxConcurrentBMCs: 2, maxConcurrentBMCs: 3}}\n", + } { + _, err := Decode([]byte(seed+addition), "/site/in.yaml") + Expect(err).To(HaveOccurred()) + } + }) + It("expands aliased merge policies without losing an explicit false override", func() { + text := strings.Replace(seed, "discovery:\n", "discovery:\n tls: &policy {insecureSkipVerify: true}\n", 1) + text = strings.ReplaceAll(text, "credentialRef: read}", "credentialRef: read, tls: {<<: *policy, insecureSkipVerify: false}}") + in, err := Decode([]byte(text), "/site/in.yaml") + Expect(err).NotTo(HaveOccurred()) + r, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + for _, target := range r.Targets { + Expect(target.Access.InsecureSkipVerify()).To(BeFalse()) + } + }) + It("keeps BMC verification enabled unless explicitly disabled and preserves presence", func() { + for _, value := range []string{"", "false", "null", "true"} { + text := seed + if value != "" { + text += " tls: {insecureSkipVerify: " + value + "}\n" + } + in, err := Decode([]byte(text), "/site/in.yaml") + Expect(err).NotTo(HaveOccurred()) + r, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + for _, target := range r.Targets { + Expect(target.Access.InsecureSkipVerify()).To(Equal(value == "true")) + Expect(target.Access.TLSConfig().InsecureSkipVerify).To(Equal(value == "true")) + } + b, err := Saved(in, &inventory.Inventory{FormatVersion: "v1alpha1"}, "/results/out.yaml") + Expect(err).NotTo(HaveOccurred()) + if value == "" { + Expect(string(b)).NotTo(ContainSubstring("insecureSkipVerify")) + } else { + Expect(string(b)).To(ContainSubstring("insecureSkipVerify: " + value)) + } + } + }) + It("isolates cached TLS policies and lets endpoint false replace global true", func() { + for _, role := range []string{"dpus", "hosts"} { + text := seed + " tls: {insecureSkipVerify: true}\n" + lines := strings.Split(text, "\n") + for i, line := range lines { + if strings.HasPrefix(line, " "+role+":") { + lines[i] = strings.Replace(line, "credentialRef: read", "credentialRef: read, tls: {insecureSkipVerify: false}", 1) + } + } + in, err := Decode([]byte(strings.Join(lines, "\n")), "/site/in.yaml") + Expect(err).NotTo(HaveOccurred()) + r, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + for _, target := range r.Targets { + Expect(target.Access.InsecureSkipVerify()).To(Equal(target.Role != role)) + clone := target.Access.TLSConfig() + clone.InsecureSkipVerify = !clone.InsecureSkipVerify + Expect(target.Access.InsecureSkipVerify()).To(Equal(target.Role != role)) + } + } + }) + It("rejects nonboolean insecure settings and contradictory CA trust", func() { + for _, value := range []string{"'true'", "1", "''"} { + _, err := Decode([]byte(seed+" tls: {insecureSkipVerify: "+value+"}\n"), "/site/in.yaml") + Expect(err).To(HaveOccurred()) + } + in, err := Decode([]byte(seed+" tls: {insecureSkipVerify: true, caFile: ca.pem}\n"), "/site/in.yaml") + Expect(err).NotTo(HaveOccurred()) + _, err = Resolve(in) + Expect(err).To(MatchError(ContainSubstring("mutually exclusive"))) + }) + It("normalizes DNS/default ports and rejects paths, userinfo and duplicate origins", func() { + origin, e := Origin("https://EXAMPLE.invalid:443/") + Expect(e).NotTo(HaveOccurred()) + Expect(origin).To(Equal("https://example.invalid")) + for _, v := range []string{"http://x", "https://u:p@x", "https://x/path", "https://x?x=y", "https://x:99999"} { + _, e := Origin(v) + Expect(e).To(HaveOccurred()) + } + in, e := Decode([]byte(strings.Replace(seed, "192.0.2.2", "192.0.2.1:443", 1)), "/site/in.yaml") + Expect(e).NotTo(HaveOccurred()) + _, e = Resolve(in) + Expect(e).To(HaveOccurred()) + }) + It("removes only one final LF or CRLF from file passwords", func() { + dir, e := os.MkdirTemp("", "dpf-config-") + Expect(e).NotTo(HaveOccurred()) + DeferCleanup(os.RemoveAll, dir) + for _, value := range []string{" space \r\n", " space \n\n", " space \r"} { + Expect(os.WriteFile(filepath.Join(dir, "password"), []byte(value), 0600)).To(Succeed()) + text := strings.Replace(seed, "env: DPF_TEST_PASSWORD", "file: password", 1) + in, e := Decode([]byte(text), filepath.Join(dir, "in.yaml")) + Expect(e).NotTo(HaveOccurred()) + r, e := Resolve(in) + Expect(e).NotTo(HaveOccurred()) + _, actual := r.Targets[0].Access.Credentials() + expected := value + if strings.HasSuffix(expected, "\n") { + expected = strings.TrimSuffix(strings.TrimSuffix(expected, "\n"), "\r") + } + Expect(actual).To(Equal(expected)) + } + }) +}) + +var _ = Describe("strict scalar regression", func() { + It("rejects boolean and numeric access fields rather than coercing strings", func() { + for _, value := range []string{"false", "123"} { + _, err := Decode([]byte(strings.Replace(seed, "username: reader", "username: "+value, 1)), "/site/in.yaml") + Expect(err).To(HaveOccurred()) + } + }) +}) diff --git a/pkg/dpf/config/scope.go b/pkg/dpf/config/scope.go new file mode 100644 index 00000000..dbf224cc --- /dev/null +++ b/pkg/dpf/config/scope.go @@ -0,0 +1,19 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package config + +import "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + +// Scope records resolved access policy without credentials or trust material. +// Trust snapshots are hashed so changed trust invalidates an earlier scope. +func Scope(t Target) inventory.ScopeEndpoint { + s := inventory.ScopeEndpoint{Role: t.Role, Endpoint: t.Endpoint, SystemURI: t.SystemURI, TLSInsecureSkipVerify: t.Access.InsecureSkipVerify()} + if t.SSH != nil { + s.SSHTrustDigest = inventory.Digest("dpf-ssh-trust-v1", t.SSH.KnownHosts()) + s.SSHSudo = t.SSH.Sudo + } else { + s.TLSTrustDigest = t.Access.TrustDigest() + } + return s +} diff --git a/pkg/dpf/config/scope_test.go b/pkg/dpf/config/scope_test.go new file mode 100644 index 00000000..4015248d --- /dev/null +++ b/pkg/dpf/config/scope_test.go @@ -0,0 +1,92 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package config + +import ( + "crypto/ed25519" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/json" + "encoding/pem" + "math/big" + "os" + "path/filepath" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("resolved Redfish trust scope", func() { + var dir string + var firstCA, secondCA []byte + BeforeEach(func() { + dir = GinkgoT().TempDir() + firstCA, secondCA = trustTestCA(1), trustTestCA(2) + Expect(os.WriteFile(filepath.Join(dir, "ca.pem"), firstCA, 0600)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "copy.pem"), firstCA, 0600)).To(Succeed()) + Expect(os.Setenv("DPF_TEST_PASSWORD", "trust-secret-canary")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_TEST_PASSWORD") + }) + resolve := func(policy string) Target { + in, err := Decode([]byte(seed+policy), filepath.Join(dir, "site.yaml")) + Expect(err).NotTo(HaveOccurred()) + r, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + return r.Targets[0] + } + It("uses CA content instead of file spelling and distinguishes all verification policies", func() { + custom := resolve(" tls: {caFile: ca.pem}\n") + copied := resolve(" tls: {caFile: copy.pem}\n") + Expect(Scope(custom)).To(Equal(Scope(copied))) + Expect(Scope(custom).TLSTrustDigest).NotTo(BeEmpty()) + Expect(os.WriteFile(filepath.Join(dir, "ca.pem"), secondCA, 0600)).To(Succeed()) + replaced := resolve(" tls: {caFile: ca.pem}\n") + Expect(Scope(replaced).TLSTrustDigest).NotTo(Equal(Scope(custom).TLSTrustDigest)) + system, insecure := resolve(""), resolve(" tls: {insecureSkipVerify: true}\n") + Expect(Scope(system).TLSTrustDigest).To(Equal("system-roots")) + Expect(Scope(insecure).TLSTrustDigest).To(Equal("verification-disabled")) + Expect(Scope(custom).TLSTrustDigest).NotTo(Equal(Scope(system).TLSTrustDigest)) + Expect(Scope(custom).TLSTrustDigest).NotTo(Equal(Scope(insecure).TLSTrustDigest)) + encoded, err := json.Marshal(Scope(custom)) + Expect(err).NotTo(HaveOccurred()) + for _, private := range []string{"trust-secret-canary", "BEGIN CERTIFICATE", "ca.pem", string(firstCA)} { + Expect(string(encoded)).NotTo(ContainSubstring(private)) + } + }) + It("keeps resolved trust immutable when files, returned CA bytes or TLS roots change", func() { + target := resolve(" tls: {caFile: ca.pem}\n") + before := Scope(target) + roots := target.Access.TLSConfig().RootCAs + Expect(os.WriteFile(filepath.Join(dir, "ca.pem"), secondCA, 0600)).To(Succeed()) + bytes := target.Access.PublicCA() + bytes[0] = 'X' + clone := target.Access.TLSConfig() + Expect(clone.RootCAs.AppendCertsFromPEM(secondCA)).To(BeTrue()) + Expect(clone.RootCAs.Equal(roots)).To(BeFalse()) + Expect(target.Access.TLSConfig().RootCAs.Equal(roots)).To(BeTrue()) + Expect(target.Access.PublicCA()).To(Equal(firstCA)) + Expect(Scope(target)).To(Equal(before)) + Expect((Access{}).TLSConfig()).To(BeNil()) + }) + It("records effective endpoint replacement policy instead of shared CA configuration", func() { + text := strings.Replace(seed, "credentialRef: read}", "credentialRef: read, tls: {}}", 1) + in, err := Decode([]byte(text+" tls: {caFile: ca.pem}\n"), filepath.Join(dir, "site.yaml")) + Expect(err).NotTo(HaveOccurred()) + r, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + Expect(Scope(r.Targets[0]).TLSTrustDigest).To(Equal("system-roots")) + Expect(Scope(r.Targets[1]).TLSTrustDigest).NotTo(Equal("system-roots")) + }) +}) + +func trustTestCA(serial int64) []byte { + public, private, err := ed25519.GenerateKey(rand.Reader) + Expect(err).NotTo(HaveOccurred()) + template := &x509.Certificate{SerialNumber: big.NewInt(serial), Subject: pkix.Name{CommonName: "Discovery trust test"}, NotBefore: time.Unix(0, 0), NotAfter: time.Unix(1<<32, 0), IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign} + der, err := x509.CreateCertificate(rand.Reader, template, template, public, private) + Expect(err).NotTo(HaveOccurred()) + return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}) +} diff --git a/pkg/dpf/config/software_test.go b/pkg/dpf/config/software_test.go new file mode 100644 index 00000000..0d25ada8 --- /dev/null +++ b/pkg/dpf/config/software_test.go @@ -0,0 +1,46 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package config + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "os" + "strings" +) + +var _ = Describe("DPF release and address source configuration", func() { + BeforeEach(func() { + Expect(os.Setenv("DPF_TEST_PASSWORD", "private")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_TEST_PASSWORD") + }) + It("defaults omitted/null releases and rejects explicit empty or unknown releases", func() { + for _, value := range []string{"", "dpf: {release: null}\n", "dpf: {release: '26.4.0'}\n"} { + in, e := Decode([]byte(value+seed), "/site/in.yaml") + Expect(e).NotTo(HaveOccurred()) + r, e := Resolve(in) + Expect(e).NotTo(HaveOccurred()) + Expect(r.Software.Version).To(Equal(releases.DefaultRelease)) + } + for _, value := range []string{"''", "'latest'", "'26.8.0'"} { + in, e := Decode([]byte("dpf: {release: "+value+"}\n"+seed), "/site/in.yaml") + Expect(e).NotTo(HaveOccurred()) + _, e = Resolve(in) + Expect(e).To(HaveOccurred()) + } + }) + It("preserves release intent and rebases optional lease source references", func() { + text := strings.Replace(seed, "discovery:\n", "discovery:\n addressLookup:\n sources: [{file: leases.txt, format: dnsmasq-leases}]\n devices: [{name: dpu1, role: arm, mac: '02:00:00:00:00:01'}]\n", 1) + in, e := Decode([]byte(text), "/site/in.yaml") + Expect(e).NotTo(HaveOccurred()) + r, e := Resolve(in) + Expect(e).NotTo(HaveOccurred()) + Expect(r.AddressLookup.Sources[0].File).To(Equal("/site/leases.txt")) + b, e := Saved(in, &inventory.Inventory{Software: r.Software}, "/output/result.yaml") + Expect(e).NotTo(HaveOccurred()) + Expect(string(b)).To(ContainSubstring("../site/leases.txt")) + Expect(string(b)).NotTo(ContainSubstring("dpf:\n")) + }) +}) diff --git a/pkg/dpf/config/ssh.go b/pkg/dpf/config/ssh.go new file mode 100644 index 00000000..88eff692 --- /dev/null +++ b/pkg/dpf/config/ssh.go @@ -0,0 +1,104 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package config + +import ( + "bytes" + "fmt" + "net" + "net/url" + "strconv" + "strings" + + "golang.org/x/crypto/ssh" +) + +// SSHAccess holds a snapshot of explicit authentication and host trust. It is +// runtime-only: saved configuration retains references, never their contents. +type SSHAccess struct { + username, password string + privateKey, knownHosts []byte + Sudo bool +} + +func (a *SSHAccess) Credentials() (string, string) { + if a == nil { + return "", "" + } + return a.username, a.password +} +func (a *SSHAccess) PrivateKey() []byte { + if a == nil { + return nil + } + return bytes.Clone(a.privateKey) +} +func (a *SSHAccess) KnownHosts() []byte { + if a == nil { + return nil + } + return bytes.Clone(a.knownHosts) +} +func (a *SSHAccess) String() string { return "" } +func (a *SSHAccess) GoString() string { return a.String() } + +// SSHOrigin accepts a host or host:port, with brackets for IPv6. It deliberately +// accepts no userinfo, path, SSH configuration alias, or proxy command. +func SSHOrigin(address string) (string, error) { + if address == "" || strings.HasSuffix(address, ":") || strings.ContainsAny(address, " \t\r\n/@?#\\") || strings.Contains(address, "://") { + return "", fmt.Errorf("expected SSH host or host:port") + } + u, err := url.Parse("ssh://" + address) + if err != nil || u.Hostname() == "" || u.User != nil || u.Path != "" { + return "", fmt.Errorf("invalid SSH address") + } + host := strings.ToLower(u.Hostname()) + if strings.Contains(host, ":") && net.ParseIP(host) == nil { + return "", fmt.Errorf("IPv6 addresses require brackets") + } + port := u.Port() + if port == "" { + port = "22" + } + n, err := strconv.Atoi(port) + if err != nil || n < 1 || n > 65535 { + return "", fmt.Errorf("invalid SSH port") + } + return "ssh://" + net.JoinHostPort(host, strconv.Itoa(n)), nil +} + +func resolveSSHAccess(c Credential, endpoint SSH, base string) (*SSHAccess, error) { + if c.Username == "" || strings.ContainsAny(c.Username, "\x00\r\n") { + return nil, fmt.Errorf("SSH credential requires a username") + } + a := &SSHAccess{username: c.Username, Sudo: endpoint.Sudo != nil && *endpoint.Sudo} + if c.PrivateKeyRef != nil { + if c.PasswordRef.Env != nil || c.PasswordRef.File != nil || c.PrivateKeyRef.File == "" { + return nil, fmt.Errorf("SSH requires exactly one passwordRef or privateKeyRef") + } + key, err := regularFile(resolvePath(base, c.PrivateKeyRef.File), 1<<20) + if err != nil { + return nil, fmt.Errorf("cannot resolve SSH private key") + } + if _, err := ssh.ParsePrivateKey(key); err != nil { + return nil, fmt.Errorf("invalid or encrypted SSH private key") + } + a.privateKey = key + } else { + password, err := resolvePassword(c.PasswordRef, base) + if err != nil { + return nil, err + } + a.password = password + } + if endpoint.KnownHostsFile == "" { + return nil, fmt.Errorf("knownHostsFile is required") + } + known, err := regularFile(resolvePath(base, endpoint.KnownHostsFile), 1<<20) + if err != nil || len(bytes.TrimSpace(known)) == 0 { + return nil, fmt.Errorf("cannot resolve nonempty SSH knownHostsFile") + } + a.knownHosts = known + return a, nil +} diff --git a/pkg/dpf/config/ssh_test.go b/pkg/dpf/config/ssh_test.go new file mode 100644 index 00000000..07d05a3a --- /dev/null +++ b/pkg/dpf/config/ssh_test.go @@ -0,0 +1,107 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package config + +import ( + "crypto/ed25519" + "crypto/rand" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "golang.org/x/crypto/ssh" + "golang.org/x/crypto/ssh/knownhosts" +) + +var _ = Describe("explicit SSH input", func() { + var dir, document string + BeforeEach(func() { + dir = GinkgoT().TempDir() + public, _, err := ed25519.GenerateKey(rand.Reader) + Expect(err).NotTo(HaveOccurred()) + key, err := ssh.NewPublicKey(public) + Expect(err).NotTo(HaveOccurred()) + Expect(os.WriteFile(filepath.Join(dir, "known_hosts"), []byte(knownhosts.Line([]string{"host.example"}, key)+"\n"), 0600)).To(Succeed()) + Expect(os.Setenv("SSH_TEST_PASSWORD", "secret-canary")).To(Succeed()) + DeferCleanup(os.Unsetenv, "SSH_TEST_PASSWORD") + document = `credentials: + reader: {username: reader, passwordRef: {env: SSH_TEST_PASSWORD}} +discovery: + hosts: + - credentialRef: reader + ssh: {address: host.example, knownHostsFile: known_hosts, sudo: false} + probes: {mode: inventory} +` + }) + It("resolves without Kubernetes or DPU endpoints, snapshots trust and preserves references", func() { + in, err := Decode([]byte(document), filepath.Join(dir, "site.yaml")) + Expect(err).NotTo(HaveOccurred()) + r, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + Expect(r.Kubeconfig).To(BeEmpty()) + Expect(r.Targets).To(HaveLen(1)) + t := r.Targets[0] + Expect(t.Endpoint).To(Equal("ssh://host.example:22")) + Expect(t.SSH.Sudo).To(BeFalse()) + _, password := t.SSH.Credentials() + Expect(password).To(Equal("secret-canary")) + raw, err := json.Marshal(t.SSH) + Expect(err).NotTo(HaveOccurred()) + Expect(string(raw)).NotTo(ContainSubstring("canary")) + Expect(fmt.Sprintf("%#v", t.SSH)).NotTo(ContainSubstring("canary")) + before := Scope(t).SSHTrustDigest + changed := t.SSH.KnownHosts() + changed[0] = 'X' + Expect(Scope(t).SSHTrustDigest).To(Equal(before)) + out, err := Saved(in, nil, filepath.Join(dir, "result", "out.yaml")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(out)).To(ContainSubstring("../known_hosts")) + Expect(string(out)).To(ContainSubstring("sudo: false")) + Expect(string(out)).NotTo(ContainSubstring("secret-canary")) + }) + It("copies SSH endpoint names without changing the trust or identity scope", func() { + in, err := Decode([]byte(document), filepath.Join(dir, "site.yaml")) + Expect(err).NotTo(HaveOccurred()) + before, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + name := "Host A" + in.Site.Discovery.Hosts[0].Name = &name + after, err := Resolve(in) + Expect(err).NotTo(HaveOccurred()) + Expect(after.Targets[0].Name).To(Equal(name)) + Expect(after.Targets[0].DisplayName()).To(Equal(`"Host A" (ssh://host.example:22)`)) + Expect(Scope(after.Targets[0])).To(Equal(Scope(before.Targets[0]))) + }) + It("rejects ambiguous transports, auth, invalid presence and management-mode SSH", func() { + for _, text := range []string{ + strings.Replace(document, "ssh:", "bmcEndpoint: https://host.example\n ssh:", 1), + strings.Replace(document, "ssh:", "tls: {insecureSkipVerify: true}\n ssh:", 1), + strings.Replace(document, "knownHostsFile: known_hosts", "knownHostsFile: ''", 1), + strings.Replace(document, "sudo: false", "sudo: 'false'", 1), + strings.Replace(document, "mode: inventory", "mode: management", 1), + strings.Replace(document, "passwordRef: {env: SSH_TEST_PASSWORD}", "passwordRef: {env: SSH_TEST_PASSWORD}, privateKeyRef: {file: key}", 1), + } { + in, err := Decode([]byte(text), filepath.Join(dir, "site.yaml")) + if err == nil { + _, err = Resolve(in) + } + Expect(err).To(HaveOccurred()) + } + }) + It("canonicalizes only bounded address forms", func() { + for input, want := range map[string]string{"HOST.EXAMPLE": "ssh://host.example:22", "host.example:2222": "ssh://host.example:2222", "[2001:db8::1]": "ssh://[2001:db8::1]:22"} { + got, err := SSHOrigin(input) + Expect(err).NotTo(HaveOccurred()) + Expect(got).To(Equal(want)) + } + for _, bad := range []string{"", "user@host", "host/path", "host:0", "host:65536", "host:", "host?x", "ssh://host", "host\ncommand"} { + _, err := SSHOrigin(bad) + Expect(err).To(HaveOccurred()) + } + }) +}) diff --git a/pkg/dpf/discovery/addresses.go b/pkg/dpf/discovery/addresses.go new file mode 100644 index 00000000..151be42f --- /dev/null +++ b/pkg/dpf/discovery/addresses.go @@ -0,0 +1,49 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/addresslookup" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "slices" + "time" +) + +func collectAddresses(r config.Request, inv *inventory.Inventory) { + if r.AddressLookup == nil { + return + } + neighbors := []addresslookup.Neighbor{} + for _, group := range [][]inventory.Device{inv.Hosts, inv.DPUs} { + for _, d := range group { + if d.Runtime == nil || d.Runtime.Network == nil { + continue + } + ref := "" + for id, e := range inv.Evidence { + if e.InputRef == d.InputRef && e.ResourceURI == "neighbors" { + ref = id + break + } + } + for _, n := range d.Runtime.Network.Neighbors { + state := "" + if slices.Contains(n.State, "REACHABLE") { + state = "REACHABLE" + } else if slices.Contains(n.State, "PERMANENT") { + state = "PERMANENT" + } + // Require the collector's authenticated provenance, not an address alone. + if ref != "" { + neighbors = append(neighbors, addresslookup.Neighbor{Address: n.Address, MAC: n.MAC, State: state, Source: d.RecordKey + "/" + n.Interface, EvidenceRef: ref}) + } + } + } + } + lookup, ev := addresslookup.Collect(*r.AddressLookup, time.Now().UTC(), config.ReadReference, neighbors) + inv.AddressLookup = lookup + for id, e := range ev { + inv.Evidence[id] = e + } +} diff --git a/pkg/dpf/discovery/auth_binding_integration_test.go b/pkg/dpf/discovery/auth_binding_integration_test.go new file mode 100644 index 00000000..d878dc0f --- /dev/null +++ b/pkg/dpf/discovery/auth_binding_integration_test.go @@ -0,0 +1,152 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "context" + "encoding/json" + "encoding/pem" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sync/atomic" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/artifacts" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +type authBindingCollector struct{} + +func (authBindingCollector) Collect(ctx context.Context, target config.Target, timeout time.Duration) (inventory.Device, map[string]inventory.Evidence, error) { + if target.Role == "dpus" { + return (redfish.Collector{}).Collect(ctx, target, timeout) + } + return (fixtureCollector{}).Collect(ctx, target, timeout) +} + +var _ = Describe("administrator session binding through publication", func() { + It("preserves a complete handoff when external or unknown authentication cannot establish privileges", func() { + var scenario atomic.Int64 + link := func(uri string) redfish.Document { return redfish.Document{"@odata.id": uri} } + collection := func(uri string) redfish.Document { + return redfish.Document{"Members@odata.count": 1, "Members": []any{link(uri)}} + } + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + const sessions = "/redfish/v1/SessionService/Sessions" + if r.Method == http.MethodPost && r.URL.Path == sessions { + w.Header().Set("X-Auth-Token", "fixture-session") + w.Header().Set("Location", sessions+"/owned") + w.WriteHeader(http.StatusCreated) + // Creation metadata is not the effective role proof; use the GET. + doc := redfish.Document{"Roles": []any{""}} + if scenario.Load() == 4 { + doc = redfish.Document{"Roles": []any{"ReadOnly"}, "UserAuthenticationSource": "ldaps://directory.example.invalid"} + } + _ = json.NewEncoder(w).Encode(doc) + return + } + if r.Method == http.MethodDelete && r.URL.Path == sessions+"/owned" { + w.WriteHeader(http.StatusNoContent) + return + } + if r.Method != http.MethodGet { + w.WriteHeader(http.StatusMethodNotAllowed) + return + } + var doc redfish.Document + switch r.URL.Path { + case "/redfish/v1/": + doc = redfish.Document{"Links": redfish.Document{"Sessions": link(sessions)}, "Systems": link("/redfish/v1/Systems"), "Chassis": link("/redfish/v1/Chassis"), "AccountService": link("/redfish/v1/AccountService")} + case sessions + "/owned": + doc = redfish.Document{"UserName": "reader"} + if scenario.Load() == 0 || scenario.Load() == 3 { + doc["Roles"] = []any{"Administrator"} + } + case "/redfish/v1/AccountService": + doc = redfish.Document{"Accounts": link("/redfish/v1/AccountService/Accounts")} + if scenario.Load() == 1 { + doc["LocalAccountAuth"] = "Disabled" + doc["LDAP"] = redfish.Document{"ServiceEnabled": true, "RemoteRoleMapping": []any{redfish.Document{"RemoteUser": "reader", "LocalRole": "ReadOnly"}}} + } else if scenario.Load() == 3 { + doc["LocalAccountAuth"] = "Fallback" + } else if scenario.Load() == 4 { + // A changed service policy does not change the session's origin. + doc["LocalAccountAuth"] = "Enabled" + } + case "/redfish/v1/AccountService/Accounts": + doc = collection("/redfish/v1/AccountService/Accounts/reader") + case "/redfish/v1/AccountService/Accounts/reader": + doc = redfish.Document{"UserName": "reader", "Enabled": true, "Locked": false, "PasswordChangeRequired": false, "RoleId": "Administrator", "Links": redfish.Document{"Role": link("/redfish/v1/AccountService/Roles/Administrator")}} + case "/redfish/v1/AccountService/Roles/Administrator": + doc = redfish.Document{"Id": "Administrator", "AssignedPrivileges": []any{"Login", "ConfigureManager", "ConfigureUsers", "ConfigureComponents", "ConfigureSelf"}} + case "/redfish/v1/Systems": + doc = collection("/redfish/v1/Systems/DPU") + case "/redfish/v1/Systems/DPU": + doc = redfish.Document{"UUID": "11111111-1111-4111-8111-111111111111", "Manufacturer": "NVIDIA", "Model": "BlueField-3"} + case "/redfish/v1/Chassis": + doc = collection("/redfish/v1/Chassis/Card") + case "/redfish/v1/Chassis/Card": + doc = redfish.Document{"Model": "BlueField-3", "SerialNumber": "CARD-ONE", "Links": redfish.Document{"ComputerSystems": []any{link("/redfish/v1/Systems/DPU")}}} + default: + w.WriteHeader(http.StatusNotFound) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(doc) + })) + DeferCleanup(server.Close) + dir, err := filepath.EvalSymlinks(GinkgoT().TempDir()) + Expect(err).NotTo(HaveOccurred()) + caPath := filepath.Join(dir, "ca.pem") + Expect(os.WriteFile(caPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw}), 0600)).To(Succeed()) + Expect(os.Setenv("DPF_AUTH_BINDING_PASSWORD", "fixture-password")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_AUTH_BINDING_PASSWORD") + seed := fmt.Sprintf("credentials:\n reader: {username: reader, passwordRef: {env: DPF_AUTH_BINDING_PASSWORD}}\ndiscovery:\n tls: {caFile: '%s'}\n hosts: [{bmcEndpoint: 'https://host.invalid', credentialRef: reader}]\n dpus: [{bmcEndpoint: '%s', credentialRef: reader}]\n probes: {mode: inventory}\n", caPath, server.URL) + inputPath := filepath.Join(dir, "seed.yaml") + Expect(os.WriteFile(inputPath, []byte(seed), 0600)).To(Succeed()) + in, err := config.Load(inputPath) + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(in) + Expect(err).NotTo(HaveOccurred()) + output := filepath.Join(dir, "discovered.yaml") + var previous []byte + for _, mode := range []int64{0, 1, 2, 3, 4} { + scenario.Store(mode) + store, err := artifacts.New(in, output, true) + Expect(err).NotTo(HaveOccurred()) + report := Collect(context.Background(), req, store.RunID(), Dependencies{BMC: authBindingCollector{}}) + Expect(report.Errors).To(BeEmpty()) + Expect(report.Inventory.DPUs[0].ObservationState).To(Equal("observed")) + Expect(report.Inventory.DPUs[0].SerialNumber).To(Equal("CARD-ONE")) + Expect(report.Inventory.Associations).To(HaveLen(1)) + Expect(store.Publish(context.Background(), &report)).To(Succeed()) + current, err := os.ReadFile(output) + Expect(err).NotTo(HaveOccurred()) + if mode == 0 { + Expect(ExitCode(report)).To(Equal(0)) + Expect(report.Discovery.Outcome).To(Equal("complete")) + previous = current + } else { + Expect(ExitCode(report)).To(Equal(2)) + Expect(report.Discovery.Outcome).To(Equal("partial")) + Expect(report.Inventory.DPUs[0].Observations["bmc-administrator"].ReasonCode).To(Equal("BMC_ADMIN_UNVERIFIABLE")) + Expect(report.Discovery.Checks).To(ContainElement(And(HaveField("ID", "dpf.management.bmc-administrator"), HaveField("State", "failed"), HaveField("ReasonCode", "BMC_ADMIN_UNVERIFIABLE"), HaveField("Message", Not(BeEmpty()))))) + Expect(current).To(Equal(previous)) + Expect(report.Artifacts).To(ContainElement(HaveField("Path", filepath.Join(store.RunDir, "partial.yaml")))) + persisted, err := os.ReadFile(filepath.Join(store.RunDir, "report.json")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(persisted)).To(ContainSubstring("BMC_ADMIN_UNVERIFIABLE")) + Expect(string(persisted)).NotTo(ContainSubstring("fixture-password")) + Expect(string(persisted)).NotTo(ContainSubstring("fixture-session")) + } + } + }) +}) diff --git a/pkg/dpf/discovery/canonical_test.go b/pkg/dpf/discovery/canonical_test.go new file mode 100644 index 00000000..a0a2dd03 --- /dev/null +++ b/pkg/dpf/discovery/canonical_test.go @@ -0,0 +1,25 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("Enriched observation ordering", func() { + It("canonicalizes unordered facts without mutating source facts", func() { + n1 := inventory.Neighbor{Address: "192.0.2.2", Interface: "eth1", State: []string{"REACHABLE", "PERMANENT"}} + n2 := inventory.Neighbor{Address: "192.0.2.1", Interface: "eth0", State: []string{"STALE"}} + r := inventory.RuntimeObservation{Network: &inventory.NetworkObservation{Neighbors: []inventory.Neighbor{n1, n2}, Links: []inventory.NetworkLink{{Name: "eth1"}, {Name: "eth0"}}}, Hardware: &inventory.HardwareObservation{Cards: []inventory.CardHardware{{PCIAddress: "0000:02:00.0", Domain: "host-pci"}, {PCIAddress: "0000:01:00.0", Domain: "host-pci"}}}} + a := canonicalRuntime(r) + Expect(r.Network.Neighbors[0].Address).To(Equal("192.0.2.2")) + Expect(r.Network.Neighbors[0].State[0]).To(Equal("REACHABLE")) + Expect(r.Hardware.Cards[0].PCIAddress).To(Equal("0000:02:00.0")) + b := r + b.Network = &inventory.NetworkObservation{Neighbors: []inventory.Neighbor{n2, n1}, Links: []inventory.NetworkLink{{Name: "eth0"}, {Name: "eth1"}}} + b.Hardware = &inventory.HardwareObservation{Cards: []inventory.CardHardware{r.Hardware.Cards[1], r.Hardware.Cards[0]}} + Expect(canonicalRuntime(b)).To(Equal(a)) + }) +}) diff --git a/pkg/dpf/discovery/discover.go b/pkg/dpf/discovery/discover.go new file mode 100644 index 00000000..2d1b58f3 --- /dev/null +++ b/pkg/dpf/discovery/discover.go @@ -0,0 +1,609 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package discovery orchestrates independent management and BMC observations. +package discovery + +import ( + "cmp" + "context" + "errors" + "maps" + "reflect" + "slices" + "strings" + "sync" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/sshprobe" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +type ProbeRunner interface { + Run(context.Context, config.Request, ManagementClient, *inventory.Inventory) ([]inventory.ProbeResult, inventory.Cleanup, error) +} +type Collector interface { + Collect(context.Context, config.Target, time.Duration) (inventory.Device, map[string]inventory.Evidence, error) +} +type Dependencies struct { + SSH Collector + BMC Collector + Probes ProbeRunner +} + +func NewReport() inventory.Report { + return inventory.Report{SchemaVersion: inventory.FormatVersion, Target: "dpf", Phase: "discover", Execution: "completed", Acceptance: "not-evaluated", Discovery: inventory.Discovery{Outcome: "invalid", Checks: []inventory.Check{}, Cleanup: inventory.Cleanup{State: "complete", Leftovers: []string{}}}, Diff: []inventory.Change{}, PendingInputs: []inventory.Problem{}, Artifacts: []inventory.Artifact{}, Errors: []inventory.Problem{}} +} +func Collect(ctx context.Context, r config.Request, runID string, deps Dependencies) inventory.Report { + started := time.Now() + logger := log.FromContext(ctx).WithName("dpf.discovery") + ctx = log.IntoContext(ctx, logger) + output := ui.FromContext(ctx) + report := NewReport() + logger.V(1).Info("DPF discovery started", "mode", r.Mode, "endpointCount", len(r.Targets)) + defer func() { + logger.V(1).Info("DPF discovery completed", "outcome", report.Discovery.Outcome, "execution", report.Execution, "cleanupState", report.Discovery.Cleanup.State, "problemCount", len(report.Errors), "duration", time.Since(started)) + }() + report.Discovery.Mode = r.Mode + inv := &inventory.Inventory{FormatVersion: inventory.FormatVersion, Collection: inventory.Collection{RunID: runID, StartedAt: time.Now().UTC(), Mode: r.Mode, Collector: "l8k-go-discovery-v1"}, RequestedScope: []inventory.ScopeEndpoint{}, DPUs: []inventory.Device{}, Hosts: []inventory.Device{}, Associations: []inventory.Association{}, Evidence: map[string]inventory.Evidence{}} + report.Inventory = inv + inv.Software = r.Software + for _, t := range r.Targets { + inv.RequestedScope = append(inv.RequestedScope, config.Scope(t)) + } + slices.SortFunc(inv.RequestedScope, func(a, b inventory.ScopeEndpoint) int { + return strings.Compare(a.Role+a.Endpoint+a.SystemURI, b.Role+b.Endpoint+b.SystemURI) + }) + ctx, cancel := context.WithTimeout(ctx, r.Timeout) + defer cancel() + var mu sync.Mutex + var wg sync.WaitGroup + problem := func(p inventory.Problem) { mu.Lock(); defer mu.Unlock(); report.Errors = append(report.Errors, p) } + var mg ManagementClient + if r.Kubeconfig != "" { + var accessErr error + mg, accessErr = ManagementAccess(r) + if accessErr != nil { + report.Execution = "failed" + report.Errors = append(report.Errors, inventory.Problem{Code: "INVALID_MANAGEMENT_ACCESS", Message: accessErr.Error(), ExitCode: 2}) + return report + } + wg.Add(1) + go func() { + defer wg.Done() + var err error + inv.Management, err = mg.Observe(ctx) + if err != nil { + problem(inventory.Problem{Code: "MANAGEMENT_INSPECTION_FAILED", Message: err.Error(), InputRef: "/management", ExitCode: 3}) + return + } + mu.Lock() + defer mu.Unlock() + for id, e := range NewEvidence(time.Now().UTC()) { + inv.Evidence[id] = e + } + }() + } + sem := make(chan struct{}, r.MaxConcurrentBMCs) + devices := make([]inventory.Device, len(r.Targets)) + for index, target := range r.Targets { + wg.Add(1) + go func(i int, t config.Target) { + defer wg.Done() + endpointLogger := logger.WithValues("endpoint", t.DisplayName(), "role", t.Role, "sourceIndex", i+1) + select { + case sem <- struct{}{}: + defer func() { <-sem }() + case <-ctx.Done(): + devices[i] = inventory.Device{RecordKey: inventory.Digest("dpf-access-v1", []string{t.Role, t.Endpoint, t.SystemURI}), InputRef: t.InputRef, BMCEndpoint: t.Endpoint, ObservationState: "unavailable", ReasonCode: "CANCELLED"} + endpointLogger.V(1).Info("Endpoint observation not started", "reasonCode", "CANCELLED") + return + } + endpointStarted := time.Now() + endpointLogger.V(1).Info("Endpoint observation started") + defer func() { + d := devices[i] + endpointLogger.V(1).Info("Endpoint observation completed", "identityState", d.ObservationState, "identityEstablished", d.ID != "", "reasonCode", d.ReasonCode, "duration", time.Since(endpointStarted)) + }() + collector := deps.BMC + if t.SSH != nil { + collector = deps.SSH + } + if collector == nil { + problem(inventory.Problem{Code: "COLLECTOR_UNAVAILABLE", Message: "selected discovery source is unavailable", InputRef: t.InputRef, ExitCode: 3}) + devices[i] = inventory.Device{RecordKey: inventory.Digest("dpf-access-v1", []string{t.Role, t.Endpoint, t.SystemURI}), InputRef: t.InputRef, ObservationState: "unavailable", ReasonCode: "COLLECTOR_UNAVAILABLE"} + return + } + d, e, err := collector.Collect(ctx, t, r.RequestTimeout) + devices[i] = d + mu.Lock() + for key, value := range e { + inv.Evidence[key] = value + } + mu.Unlock() + if err != nil { + if t.SSH != nil { + code := d.ReasonCode + if code == "" { + code = "SSH_OBSERVATION_FAILED" + } + stage := "observation" + tool := "" + var diagnostic *sshprobe.Error + if errors.As(err, &diagnostic) { + stage = diagnostic.Stage + tool = diagnostic.Tool + } + problem(inventory.Problem{Code: code, Message: sshprobe.Reason(err), Tool: tool, InputRef: t.InputRef, Stage: stage, ExitCode: 3}) + } else { + for _, diagnostic := range redfish.Diagnostics(err) { + problem(inventory.Problem{Code: diagnostic.Code, Message: "Redfish operation failed", InputRef: t.InputRef, Method: diagnostic.Method, ResourceURI: diagnostic.URI, HTTPStatus: diagnostic.Status, Stage: diagnostic.Stage, ExitCode: 3}) + } + } + if d.ReasonCode == "CLEANUP_INCOMPLETE" { + mu.Lock() + report.Discovery.Cleanup.State = "incomplete" + report.Discovery.Cleanup.Leftovers = append(report.Discovery.Cleanup.Leftovers, t.Endpoint+redfish.CleanupReceipt(err)+": session cleanup incomplete") + mu.Unlock() + } + } + }(index, target) + } + wg.Wait() + for i, d := range devices { + if r.Targets[i].Role == "dpus" { + inv.DPUs = append(inv.DPUs, d) + } else { + inv.Hosts = append(inv.Hosts, d) + } + } + requestedDPUs := false + collectAddresses(r, inv) + for _, t := range r.Targets { + requestedDPUs = requestedDPUs || t.Role == "dpus" + } + if !requestedDPUs { + inv.DPUs = HostCards(inv.Hosts) + } + output.Info("[RUNNING] Host/DPU physical association") + inv.Associations, report.Discovery.Checks = Correlate(inv.DPUs, inv.Hosts) + if len(inv.Associations) == len(inv.DPUs) && len(inv.DPUs) > 0 { + output.Info("[PASS] Host/DPU physical association: %d unique matches", len(inv.Associations)) + } else { + output.Info("[FAIL] Host/DPU physical association: %d/%d uniquely matched", len(inv.Associations), len(inv.DPUs)) + } + selected := map[string]bool{} + for _, a := range inv.Associations { + selected[a.HostAdapterKey] = true + } + for hi := range inv.Hosts { + for ai := range inv.Hosts[hi].Adapters { + if selected[inv.Hosts[hi].Adapters[ai].Key] { + inv.Hosts[hi].Adapters[ai].Selection = "selected" + } + } + } + addCheck := func(id, subject string, ok bool, reason string, refs []string, phase string) { + state := "failed" + if ok { + state = "passed" + } + report.Discovery.Checks = append(report.Discovery.Checks, inventory.Check{ID: id, Subject: subject, Vantage: "executor", RequiredPhase: phase, State: state, ReasonCode: reason, EvidenceRefs: refs}) + } + if inv.Management != nil { + cp := 0 + for _, n := range inv.Management.Nodes { + if n.ControlPlane { + cp++ + } + } + phase := "generate" + if r.Mode == "management" { + phase = "discover" + } + addCheck("dpf.management.control-plane-count", "management", cp == 3, "EXACTLY_THREE_CONTROL_PLANES", inv.Management.EvidenceRefs, phase) + if r.Mode == "management" && cp == 3 { + if deps.Probes == nil { + report.Errors = append(report.Errors, inventory.Problem{Code: "PROBE_ASSETS_UNAVAILABLE", Message: "management probes require qualified installed probe assets", ExitCode: 2}) + } else { + rows, cleanup, err := deps.Probes.Run(ctx, r, mg, inv) + inv.Probes = rows + if cleanup.State != "complete" { + report.Discovery.Cleanup.State = cleanup.State + } + report.Discovery.Cleanup.Leftovers = append(report.Discovery.Cleanup.Leftovers, cleanup.Leftovers...) + if err != nil { + report.Errors = append(report.Errors, inventory.Problem{Code: "PROBE_FAILED", Message: err.Error(), ExitCode: 3}) + } + } + } + fresh, err := mg.Observe(ctx) + if err != nil { + report.Errors = append(report.Errors, inventory.Problem{Code: "MANAGEMENT_RECHECK_FAILED", Message: "management identity recheck unavailable", InputRef: "/management", ExitCode: 3}) + } else if fresh.SystemNamespaceUID != inv.Management.SystemNamespaceUID || !sameNodes(fresh.Nodes, inv.Management.Nodes) { + report.Errors = append(report.Errors, inventory.Problem{Code: "MANAGEMENT_SCOPE_CHANGED", Message: "management identity or node membership changed during collection", ExitCode: 3}) + } + } + output.Info("[RUNNING] Required discovery checks") + report.Discovery.Checks = append(report.Discovery.Checks, ProbeChecks(inv, r.Targets)...) + for _, d := range append(slices.Clone(inv.DPUs), inv.Hosts...) { + addCheck("dpf.device.identity", d.RecordKey, d.ID != "" && d.ObservationState == "observed", "PHYSICAL_IDENTITY_REQUIRED", d.EvidenceRefs, "discover") + addCheck("dpf.device.qualification", d.RecordKey, d.Qualification == "qualified" && d.InventoryCompleteness == "complete", "QUALIFIED_ADAPTER_REQUIRED", d.EvidenceRefs, "generate") + } + addCheck("dpf.inventory.dpus", "inventory", len(inv.DPUs) > 0, "DPU_IDENTITY_REQUIRED", nil, "discover") + for i := range inv.DPUs { + d := &inv.DPUs[i] + admin := d.Observations["bmc-administrator"] + if admin.State == "" || d.BMCEndpoint == "" || d.SSHEndpoint != "" { + admin = inventory.Observation{State: "failed", ReasonCode: "BMC_ADMIN_NOT_VERIFIED", Message: "An explicit DPU BMC endpoint with verified administrator privileges is required; host or ARM SSH identity cannot substitute for BMC authorization."} + } + if d.Observations == nil { + d.Observations = map[string]inventory.Observation{} + } + d.Observations["bmc-administrator"] = admin + adminState := "failed" + if admin.State == "observed" && admin.ReasonCode == "BMC_ADMIN_VERIFIED" && d.BMCEndpoint != "" && d.SSHEndpoint == "" { + adminState = "passed" + } + report.Discovery.Checks = append(report.Discovery.Checks, inventory.Check{ID: "dpf.management.bmc-administrator", Subject: d.RecordKey, Vantage: "executor", RequiredPhase: "discover", State: adminState, ReasonCode: admin.ReasonCode, Message: admin.Message, EvidenceRefs: d.EvidenceRefs}) + state, reason := "not-evaluated", "REDFISH_AUTHORIZATION_NOT_EVALUATED" + if d.Authentication.PasswordRead == "verified" { + state, reason = "passed", "INVENTORY_READ_VERIFIED" + } + report.Discovery.Checks = append(report.Discovery.Checks, inventory.Check{ID: "dpf.management.redfish-read", Subject: d.RecordKey, RequiredPhase: "deploy", State: state, ReasonCode: reason, EvidenceRefs: d.EvidenceRefs}) + report.Discovery.Checks = append(report.Discovery.Checks, inventory.Check{ID: "dpf.management.dpf-authentication", Subject: d.RecordKey, RequiredPhase: "deploy", State: "not-evaluated", ReasonCode: "DPF_AUTHENTICATION_NOT_EVALUATED", EvidenceRefs: []string{}}) + } + report.Discovery.Checks = append(report.Discovery.Checks, SoftwareChecks(inv)...) + for i := range report.Discovery.Checks { + check := &report.Discovery.Checks[i] + ExplainCheck(check) + checkLogger := logger + for index, device := range devices { + if check.Subject != "" && (check.Subject == device.RecordKey || check.Subject == device.ID) { + checkLogger = logger.WithValues("endpoint", r.Targets[index].DisplayName(), "role", r.Targets[index].Role, "sourceIndex", index+1) + break + } + } + checkLogger.V(1).Info("Discovery check evaluated", "check", check.ID, "state", check.State, "requiredPhase", check.RequiredPhase, "reasonCode", check.ReasonCode) + } + report.Discovery.Outcome = "complete" + for _, c := range report.Discovery.Checks { + if c.RequiredPhase == "discover" && c.State != "passed" { + report.Discovery.Outcome = "partial" + } + } + if len(report.Errors) > 0 || report.Discovery.Cleanup.State != "complete" { + report.Discovery.Outcome = "partial" + } + if ctx.Err() != nil { + report.Discovery.Outcome = "partial" + report.Execution = "failed" + code := 3 + if ctx.Err() == context.Canceled { + report.Execution = "cancelled" + code = 1 + } + report.Errors = append(report.Errors, inventory.Problem{Code: "COLLECTION_INTERRUPTED", Message: "discovery cancelled or timed out", ExitCode: code}) + } + inv.Collection.FinishedAt = time.Now().UTC() + Canonicalize(inv) + if r.Kubeconfig != "" && inv.Management == nil { + inv.ScopeDigest = "" + } + report.Diff = Diff(r.Input.Site.Inventory, inv) + if scopeConflict(r.Input.Site.Inventory, inv) { + report.Discovery.Outcome = "partial" + message := "resolved scope changed; use a fresh seed and new output" + if inventory.LegacyHostCardBindings(r.Input.Site.Inventory) { + message = "saved inventory uses legacy Redfish card bindings that cannot protect refresh; verify endpoint ownership and use a fresh seed without inventory and a new output" + } + report.Errors = append(report.Errors, inventory.Problem{Code: "SCOPE_CONFLICT", Message: message, ExitCode: 2}) + } + report.PendingInputs = []inventory.Problem{{Code: "LATER_PHASE_INPUT", Message: "DPU API allocation and selected service profile are evaluated by future generation", InputRef: "/networks/dpuAPI"}} + slices.SortFunc(report.Errors, func(a, b inventory.Problem) int { return strings.Compare(a.InputRef+a.Code, b.InputRef+b.Code) }) + if report.Discovery.Outcome == "complete" { + output.Info("[PASS] Required discovery checks completed; publication pending") + } else { + output.Info("[FAIL] Required discovery checks: incomplete evidence; complete handoff will not be published") + } + return report +} +func scopeConflict(old, current *inventory.Inventory) bool { + return old != nil && ((old.ScopeDigest != "" && current.ScopeDigest != "" && old.ScopeDigest != current.ScopeDigest) || inventory.IdentityConflict(old, current)) +} +func sameNodes(a, b []inventory.Node) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i].UID != b[i].UID || a[i].Name != b[i].Name { + return false + } + } + return true +} +func Correlate(dpus, hosts []inventory.Device) ([]inventory.Association, []inventory.Check) { + associations := []inventory.Association{} + checks := []inventory.Check{} + seenIDs := map[string]int{} + for _, d := range append(slices.Clone(dpus), hosts...) { + if d.ID != "" { + seenIDs[d.ID]++ + } + } + for _, d := range dpus { + matches := []inventory.Association{} + vendorConflict := false + refs := slices.Clone(d.EvidenceRefs) + for hi := range hosts { + h := &hosts[hi] + for ai := range h.Adapters { + a := &h.Adapters[ai] + if d.ObservationState == "observed" && h.ObservationState == "observed" && d.SerialNumber != "" && a.SerialNumber == d.SerialNumber { + // Selection annotations are applied by the orchestrator after correlation. + matches = append(matches, inventory.Association{DPUID: d.ID, HostID: h.ID, HostAdapterKey: a.Key, SerialNumber: d.SerialNumber, EvidenceRefs: append(slices.Clone(d.EvidenceRefs), a.EvidenceRefs...)}) + // Readers normalize known PCI vendors to four-digit lowercase hex. + // Keep contradictory candidates: dropping one could turn ambiguity + // into an apparently unique NVIDIA BlueField association. + if a.VendorID != "" && a.VendorID != "15b3" { + vendorConflict = true + refs = append(refs, a.EvidenceRefs...) + } + } + } + } + state := "failed" + reason := "UNIQUE_PHYSICAL_MATCH_REQUIRED" + if vendorConflict { + reason = "HOST_ADAPTER_VENDOR_CONFLICT" + } else if len(matches) == 1 && d.ID != "" && matches[0].HostID != "" && seenIDs[d.ID] == 1 && seenIDs[matches[0].HostID] == 1 { + state = "passed" + associations = append(associations, matches[0]) + } + checks = append(checks, inventory.Check{ID: "dpf.inventory.association", Subject: d.RecordKey, Vantage: "executor", RequiredPhase: "discover", State: state, ReasonCode: reason, EvidenceRefs: refs}) + } + return associations, checks +} +func Canonicalize(inv *inventory.Inventory) { + sortDevices := func(devices []inventory.Device) { + slices.SortFunc(devices, func(a, b inventory.Device) int { return strings.Compare(a.RecordKey, b.RecordKey) }) + for i := range devices { + slices.SortFunc(devices[i].Adapters, func(a, b inventory.Adapter) int { return strings.Compare(a.Key, b.Key) }) + slices.SortFunc(devices[i].Firmware, func(a, b inventory.Firmware) int { + return strings.Compare(a.Component+a.RawVersion, b.Component+b.RawVersion) + }) + } + } + sortDevices(inv.DPUs) + sortDevices(inv.Hosts) + slices.SortFunc(inv.Associations, func(a, b inventory.Association) int { return strings.Compare(a.DPUID, b.DPUID) }) + inv.Fingerprint = inventory.Digest("dpf-facts-v1", material(inv)) + // A loaded v3 inventory retains only the digest. Fresh collections supply + // RequestedScope; do not replace a persisted guard with an empty scope. + if inv.RequestedScope == nil && inv.ScopeDigest != "" { + return + } + inv.ScopeDigest = inventory.Digest("dpf-infrastructure-scope-v1", struct { + Mode string + Endpoints []inventory.ScopeEndpoint + }{inv.Collection.Mode, inv.RequestedScope}) + if inv.Management != nil { + inv.ScopeDigest = inventory.Digest("dpf-scope-v1", struct { + API, UID, Trust, Mode string + Endpoints []inventory.ScopeEndpoint + }{inv.Management.APIEndpoint, inv.Management.SystemNamespaceUID, inv.Management.TrustDigest, inv.Collection.Mode, inv.RequestedScope}) + } +} + +// Dedicated material projection excludes timestamps, readiness, credentials and aliases. +func material(inv *inventory.Inventory) map[string]any { + fields := map[string]any{} + if inv.Software != nil { + fields["software"] = releases.Summarize(*inv.Software) + } + if inv.AddressLookup != nil { + fields["addressLookup"] = materialAddressLookup(*inv.AddressLookup) + } + if inv.Management != nil { + fields["management.identity"] = []string{inv.Management.APIEndpoint, inv.Management.SystemNamespaceUID, inv.Management.TrustDigest, inv.Management.KubernetesVersion} + for _, n := range inv.Management.Nodes { + // Empty observed facts have one material form across YAML/JSON codecs. + addresses := append([]string(nil), n.Addresses...) + slices.Sort(addresses) + interfaces := append([]inventory.Interface(nil), n.Interfaces...) + for i := range interfaces { + interfaces[i].Addresses = append([]string(nil), interfaces[i].Addresses...) + slices.Sort(interfaces[i].Addresses) + } + slices.SortFunc(interfaces, func(a, b inventory.Interface) int { + return cmp.Or(cmp.Compare(a.Index, b.Index), strings.Compare(a.Name, b.Name)) + }) + routes := append([]inventory.Route(nil), n.Routes...) + slices.SortFunc(routes, func(a, b inventory.Route) int { + return cmp.Or(cmp.Compare(a.Table, b.Table), strings.Compare(a.Destination, b.Destination), strings.Compare(a.Gateway, b.Gateway), cmp.Compare(a.InterfaceIndex, b.InterfaceIndex)) + }) + fields["management.node."+n.UID] = []any{n.Name, n.OS, n.Architecture, addresses, interfaces, routes} + + } + } + for _, group := range []struct { + role string + devices []inventory.Device + }{{"dpus", inv.DPUs}, {"hosts", inv.Hosts}} { + for _, d := range group.devices { + key := "device." + d.RecordKey + if d.Runtime != nil { + runtime := canonicalRuntime(*d.Runtime) + d.Runtime = &runtime + } + w, err := inventory.DeviceToOutput(d, group.role) + if err != nil { + // Publication rejects malformed facts. Keep them visible in the + // comparison as well, rather than claiming an unchanged record. + fields[key+".invalid"] = err.Error() + continue + } + fields[key+".identity"] = []string{w.ID, w.SerialNumber, w.SystemUUID, w.Manufacturer, w.Model, w.Method} + fields[key+".endpoints"] = []string{w.SSHEndpoint, w.BMCEndpoint, w.SystemURI} + fields[key+".psid"] = w.PSID + fields[key+".bmcAdministratorVerified"] = w.BMCAdministratorVerified + fields[key+".firmwareInventoryComplete"] = w.FirmwareInventoryComplete + fields[key+".firmware"] = w.Firmware + fields[key+".runtime"] = w.Runtime + fields[key+".advertisedResetTypes"] = canonicalEmptySlice(w.AdvertisedResetTypes) + for _, a := range w.Adapters { + a.PCIFunctions = canonicalFunctions(a.PCIFunctions) + fields[key+".adapter."+a.Key] = a + } + for _, issue := range w.Issues { + fields[key+".issue."+issue.Field+"."+issue.Code] = issue.Tool + } + } + } + for _, a := range inv.Associations { + fields["association."+a.DPUID] = []string{a.HostID, a.HostAdapterKey, a.SerialNumber} + } + return fields +} + +// materialAddressLookup preserves candidate facts while excluding provenance +// references, which are deliberately omitted from the saved YAML config. +func materialAddressLookup(lookup inventory.AddressLookup) inventory.AddressLookup { + lookup.Candidates = slices.Clone(lookup.Candidates) + for i := range lookup.Candidates { + lookup.Candidates[i].EvidenceRef = "" + } + if len(lookup.Candidates) == 0 { + lookup.Candidates = nil + } + lookup.Observations = maps.Clone(lookup.Observations) + for key, observation := range lookup.Observations { + // Values are locators already represented by Candidates or transient + // source-byte evidence digests. Neither belongs in material comparison. + observation.Values = nil + lookup.Observations[key] = observation + } + if len(lookup.Observations) == 0 { + lookup.Observations = nil + } + return lookup +} + +func Diff(old, current *inventory.Inventory) []inventory.Change { + out := []inventory.Change{} + if old == nil { + return append(out, inventory.Change{Path: "inventory", State: "baseline-absent", RequiredPhase: "generate"}) + } + before, after := material(old), material(current) + keys := []string{} + for k := range before { + keys = append(keys, k) + } + for k := range after { + if _, ok := before[k]; !ok { + keys = append(keys, k) + } + } + slices.Sort(keys) + for _, k := range keys { + a, aok := before[k] + b, bok := after[k] + if reflect.DeepEqual(a, b) { + continue + } + state := "changed" + if !aok { + state = "added" + } + if !bok { + state = "unknown" + } + for _, group := range [][]inventory.Device{current.DPUs, current.Hosts} { + for _, d := range group { + if d.ObservationState != "observed" && strings.HasPrefix(k, "device."+d.RecordKey+".") { + state = "unknown" + } + } + } + out = append(out, inventory.Change{Path: k, State: state, Previous: a, Current: b, RequiredPhase: "generate"}) + } + return out +} +func ExitCode(r inventory.Report) int { + code := 0 + for _, e := range r.Errors { + if e.ExitCode == 1 { + return 1 + } + if e.ExitCode == 3 { + code = 3 + } else if code == 0 { + code = e.ExitCode + } + } + if code != 0 { + return code + } + if r.Discovery.Outcome != "complete" { + return 2 + } + return 0 +} + +// ProbeChecks derives required rows from frozen node/endpoint scope, never aggregate counts. +func ProbeChecks(inv *inventory.Inventory, targets []config.Target) []inventory.Check { + if inv.Collection.Mode != "management" { + return []inventory.Check{{ID: "dpf.management.path", Subject: "management", RequiredPhase: "generate", State: "not-requested", ReasonCode: "INVENTORY_MODE", EvidenceRefs: []string{}}} + } + checks := []inventory.Check{} + if inv.Management == nil { + return checks + } + destinations := []string{"management-api"} + for _, t := range targets { + if t.Role == "dpus" { + destinations = append(destinations, inventory.Digest("probe-destination-v1", t.Endpoint)) + } + } + rows := map[string][]inventory.ProbeResult{} + for _, r := range inv.Probes { + key := r.NodeUID + "/" + r.NetworkMode + "/" + r.Destination + rows[key] = append(rows[key], r) + } + for _, n := range inv.Management.Nodes { + for _, mode := range []string{"pod-network", "node-network"} { + for _, destination := range destinations { + key := n.UID + "/" + mode + "/" + destination + state := "unknown" + refs := []string{} + values := rows[key] + if len(values) == 1 { + v := values[0] + if v.EvidenceRef != "" { + if _, ok := inv.Evidence[v.EvidenceRef]; ok { + refs = append(refs, v.EvidenceRef) + } + } + state = "passed" + if len(refs) == 0 || len(v.Addresses) == 0 || v.DNS == "unavailable" { + state = "unknown" + } + for _, a := range v.Addresses { + if a.TCP != "reachable" || a.TLS != "verified" { + state = "failed" + } + } + } else if len(values) > 1 { + state = "failed" + } + checks = append(checks, inventory.Check{ID: "dpf.management.path", Subject: key, Vantage: n.UID + "/" + mode, RequiredPhase: "discover", State: state, ReasonCode: "EXACT_NETWORK_COVERAGE_REQUIRED", EvidenceRefs: refs}) + } + } + } + return checks +} diff --git a/pkg/dpf/discovery/discover_test.go b/pkg/dpf/discovery/discover_test.go new file mode 100644 index 00000000..85b4736d --- /dev/null +++ b/pkg/dpf/discovery/discover_test.go @@ -0,0 +1,403 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "context" + "encoding/json" + "maps" + "os" + "slices" + "testing" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/addresslookup" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +func TestDiscovery(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "DPF discovery checks") } +func estate() inventory.Inventory { + return inventory.Inventory{Collection: inventory.Collection{Mode: "inventory"}, Management: &inventory.Management{APIEndpoint: "https://api.invalid", SystemNamespaceUID: "cluster-a", TrustDigest: "system-roots", KubernetesVersion: "v1", Nodes: []inventory.Node{{Name: "cp", UID: "node-a", Ready: true}}}, RequestedScope: []inventory.ScopeEndpoint{}, DPUs: []inventory.Device{{ID: "dpu-a", BMCEndpoint: "https://dpu-a.invalid", Model: "B4240", ReaderAdapter: "bluefield-card-v1", RecordKey: "a", SerialNumber: "CARD-A", ObservationState: "observed"}, {ID: "dpu-b", BMCEndpoint: "https://dpu-b.invalid", Model: "B4240", ReaderAdapter: "bluefield-card-v1", RecordKey: "b", SerialNumber: "CARD-B", ObservationState: "observed"}}, Hosts: []inventory.Device{{ObservationState: "observed", ID: "host-a", BMCEndpoint: "https://host.invalid", RecordKey: "host", Adapters: []inventory.Adapter{{Key: "adapter-a", SerialNumber: "CARD-A"}, {Key: "adapter-b", SerialNumber: "CARD-B"}, {Key: "adapter-c", SerialNumber: "CARD-C", Selection: "not-selected"}}}}, Associations: []inventory.Association{}} +} + +var _ = Describe("inventory semantics", func() { + It("maps two selected cards to one host without selecting the extra or mutating input", func() { + v := estate() + before, _ := json.Marshal(v.Hosts) + a, c := Correlate(v.DPUs, v.Hosts) + Expect(a).To(HaveLen(2)) + for _, row := range c { + Expect(row.State).To(Equal("passed")) + } + after, _ := json.Marshal(v.Hosts) + Expect(after).To(Equal(before)) + Expect(v.Hosts[0].Adapters[2].Selection).To(Equal("not-selected")) + }) + It("rejects duplicate endpoint physical identity and ambiguous host matches", func() { + v := estate() + v.Hosts = append(v.Hosts, v.Hosts[0]) + a, c := Correlate(v.DPUs, v.Hosts) + Expect(a).To(BeEmpty()) + Expect(c[0].State).To(Equal("failed")) + }) + It("accepts NVIDIA or unknown adapter vendors without filtering mixed host inventory", func() { + for _, vendor := range []string{"15b3", ""} { + v := estate() + v.Hosts[0].Adapters[0].VendorID = vendor + v.Hosts[0].Adapters[2].VendorID = "8086" + before, err := json.Marshal(v.Hosts) + Expect(err).NotTo(HaveOccurred()) + associations, checks := Correlate(v.DPUs, v.Hosts) + Expect(associations).To(HaveLen(2)) + Expect(checks).To(ConsistOf(HaveField("State", "passed"), HaveField("State", "passed"))) + after, err := json.Marshal(v.Hosts) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(before)) + } + }) + It("rejects a same-serial PCI vendor contradiction with both source evidence references", func() { + v := estate() + v.DPUs = v.DPUs[:1] + v.DPUs[0].EvidenceRefs = []string{"dpu-serial"} + v.Hosts[0].Adapters[0].VendorID = "8086" + v.Hosts[0].Adapters[0].EvidenceRefs = []string{"host-serial", "host-pci-vendor"} + before, err := json.Marshal(v) + Expect(err).NotTo(HaveOccurred()) + associations, checks := Correlate(v.DPUs, v.Hosts) + Expect(associations).To(BeEmpty()) + Expect(checks).To(HaveLen(1)) + Expect(checks[0].State).To(Equal("failed")) + Expect(checks[0].ReasonCode).To(Equal("HOST_ADAPTER_VENDOR_CONFLICT")) + Expect(checks[0].EvidenceRefs).To(ConsistOf("dpu-serial", "host-serial", "host-pci-vendor")) + ExplainCheck(&checks[0]) + Expect(checks[0].Message).To(ContainSubstring("Verify the selected endpoints")) + after, err := json.Marshal(v) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(before)) + }) + It("does not discard a contradictory candidate to accept another same-serial NVIDIA card", func() { + for _, reverse := range []bool{false, true} { + v := estate() + v.DPUs = v.DPUs[:1] + v.Hosts[0].Adapters[0].VendorID = "8086" + other := inventory.Device{ID: "host-b", RecordKey: "other-host", ObservationState: "observed", Adapters: []inventory.Adapter{{Key: "nvidia-card", SerialNumber: "CARD-A", VendorID: "15b3"}}} + if reverse { + v.Hosts = append([]inventory.Device{other}, v.Hosts...) + } else { + v.Hosts = append(v.Hosts, other) + } + associations, checks := Correlate(v.DPUs, v.Hosts) + Expect(associations).To(BeEmpty()) + Expect(checks[0].ReasonCode).To(Equal("HOST_ADAPTER_VENDOR_CONFLICT")) + Expect(checks[0].State).To(Equal("failed")) + } + }) + It("keeps fingerprints stable across order/time/readiness and changes them for identity", func() { + a := estate() + Canonicalize(&a) + first := a.Fingerprint + a.Collection.StartedAt = time.Now() + a.Management.Nodes[0].Ready = false + a.DPUs[0], a.DPUs[1] = a.DPUs[1], a.DPUs[0] + Canonicalize(&a) + Expect(a.Fingerprint).To(Equal(first)) + a.DPUs[0].ID = "replacement" + Canonicalize(&a) + Expect(a.Fingerprint).NotTo(Equal(first)) + }) + It("does not treat removal of legacy BMC metadata as a material or scope change", func() { + before, after := estate(), estate() + before.Evidence = map[string]inventory.Evidence{"old-bmc-ref": {Kind: "redfish", Authenticated: true}} + before.DPUs[0].EvidenceRefs = []string{"old-bmc-ref"} + before.Hosts[0].EvidenceRefs = []string{"old-bmc-ref"} + before.Hosts[0].Adapters[0].EvidenceRefs = []string{"old-bmc-ref"} + before.DPUs[0].Firmware = []inventory.Firmware{{Component: "bmc", State: "known", RawVersion: "1.0", Activation: "unknown", EvidenceRefs: []string{"old-bmc-ref"}}} + after.DPUs[0].Firmware = []inventory.Firmware{{Component: "bmc", State: "known", RawVersion: "1.0", Activation: "unknown"}} + before.Associations, _ = Correlate(before.DPUs, before.Hosts) + after.Associations, _ = Correlate(after.DPUs, after.Hosts) + Canonicalize(&before) + Canonicalize(&after) + Expect(after.Fingerprint).To(Equal(before.Fingerprint)) + Expect(after.ScopeDigest).To(Equal(before.ScopeDigest)) + Expect(Diff(&before, &after)).To(BeEmpty()) + }) + It("keeps an unchanged SSH optional-read failure stable through v3 save and reload", func() { + before := estate() + before.Management.Nodes[0].Addresses = []string{} + before.Hosts[0].SSHEndpoint = "ssh://192.0.2.10:22" + before.Hosts[0].ReaderAdapter = "linux-host-ssh-v1" + before.Hosts[0].Runtime = &inventory.RuntimeObservation{Observations: map[string]inventory.Observation{ + "routes": {State: "unavailable", Values: []string{}, ReasonCode: "COMMAND_FAILED", Message: "ip route failed", Tool: "ip"}, + }} + Canonicalize(&before) + + encoded, err := yaml.Marshal(before) + Expect(err).NotTo(HaveOccurred()) + var after inventory.Inventory + Expect(yaml.Unmarshal(encoded, &after)).To(Succeed()) + Canonicalize(&after) + + failure := after.Hosts[0].Runtime.Observations["routes"] + Expect(failure.State).To(Equal("unavailable")) + Expect(failure.Values).To(BeEmpty()) + Expect(failure.ReasonCode).To(Equal("COMMAND_FAILED")) + Expect(failure.Message).To(Equal("ip route failed")) + Expect(failure.Tool).To(Equal("ip")) + Expect(Diff(&before, &after)).To(BeEmpty()) + Expect(after.Fingerprint).To(Equal(before.Fingerprint)) + }) + It("retains SSH host references when correlating with a BMC DPU without metadata", func() { + v := estate() + v.Hosts[0].SSHEndpoint = "ssh://192.0.2.1:22" + v.Hosts[0].Adapters[0].EvidenceRefs = []string{"ssh-host-card"} + v.DPUs[0].BMCEndpoint = "https://192.0.2.2" + associations, checks := Correlate(v.DPUs, v.Hosts) + Expect(associations).To(ContainElement(And(HaveField("DPUID", "dpu-a"), HaveField("EvidenceRefs", []string{"ssh-host-card"})))) + Expect(checks).To(ContainElement(And(HaveField("Subject", "a"), HaveField("State", "passed")))) + }) + + It("fingerprints Redfish PSID values and eligibility without diagnostic prose", func() { + before, after := estate(), estate() + before.DPUs[0].Observations = map[string]inventory.Observation{"psid": {State: "observed", Values: []string{"MT_0000001774"}, ReasonCode: "PSID_OBSERVED", Tool: "redfish", Message: "initial"}} + after.DPUs[0].Observations = map[string]inventory.Observation{"psid": {State: "observed", Values: []string{"MT_0000001774"}, ReasonCode: "PSID_OBSERVED", Tool: "redfish", Message: "different prose"}} + Canonicalize(&before) + Canonicalize(&after) + Expect(after.Fingerprint).To(Equal(before.Fingerprint)) + after.DPUs[0].Observations["psid"] = inventory.Observation{State: "observed", Values: []string{"MT_0000001775"}, ReasonCode: "PSID_OBSERVED", Tool: "redfish"} + Canonicalize(&after) + Expect(after.Fingerprint).NotTo(Equal(before.Fingerprint)) + after.DPUs[0].Observations["psid"] = inventory.Observation{State: "unavailable", ReasonCode: "PSID_CONFLICT", Tool: "redfish"} + Canonicalize(&after) + Expect(after.Fingerprint).NotTo(Equal(before.Fingerprint)) + Expect(Diff(&before, &after)).To(ContainElement(HaveField("Path", ContainSubstring(".psid")))) + }) + It("blocks endpoint replacement but preserves unknown observations", func() { + old := estate() + cur := estate() + cur.DPUs[0].ID = "replacement" + Expect(inventory.IdentityConflict(&old, &cur)).To(BeTrue()) + cur.DPUs[0].ID = "" + Expect(inventory.IdentityConflict(&old, &cur)).To(BeFalse()) + Expect(Diff(nil, &cur)[0].State).To(Equal("baseline-absent")) + }) + It("applies error precedence without hiding inspection failures", func() { + r := NewReport() + r.Discovery.Outcome = "partial" + Expect(ExitCode(r)).To(Equal(2)) + r.Errors = []inventory.Problem{{ExitCode: 2}, {ExitCode: 3}} + Expect(ExitCode(r)).To(Equal(3)) + r.Errors = append(r.Errors, inventory.Problem{ExitCode: 1}) + Expect(ExitCode(r)).To(Equal(1)) + }) +}) + +var _ = Describe("refresh regressions", func() { + It("canonicalizes nested network ordering without changing the observations", func() { + a := estate() + a.Management.Nodes[0].Interfaces = []inventory.Interface{{Index: 2, Name: "b", Addresses: []string{"192.0.2.2/24", "192.0.2.1/24"}}, {Index: 1, Name: "a"}} + a.Management.Nodes[0].Routes = []inventory.Route{{Destination: "b", Table: 2}, {Destination: "a", Table: 1}} + Canonicalize(&a) + original := a.Fingerprint + b := estate() + b.Management.Nodes[0].Interfaces = []inventory.Interface{{Index: 1, Name: "a"}, {Index: 2, Name: "b", Addresses: []string{"192.0.2.1/24", "192.0.2.2/24"}}} + b.Management.Nodes[0].Routes = []inventory.Route{{Destination: "a", Table: 1}, {Destination: "b", Table: 2}} + Canonicalize(&b) + Expect(b.Fingerprint).To(Equal(original)) + Expect(a.Management.Nodes[0].Interfaces[0].Name).To(Equal("b")) + }) + It("classifies failed current identity observations as unknown", func() { + old := estate() + current := estate() + current.DPUs[0].ObservationState = "unavailable" + current.DPUs[0].ID = "" + changes := Diff(&old, ¤t) + Expect(changes).NotTo(BeEmpty()) + for _, change := range changes { + if change.Path == "device.a.identity" { + Expect(change.State).To(Equal("unknown")) + } + } + }) +}) + +var _ = Describe("scope evidence on refresh", func() { + It("rejects changed or legacy Redfish trust policies without calling unread observations replacements", func() { + for _, prior := range []string{"", "system-roots", "custom-ca-before", "verification-disabled"} { + old := estate() + old.RequestedScope = []inventory.ScopeEndpoint{{Role: "dpus", Endpoint: "https://192.0.2.1", TLSTrustDigest: prior}} + Canonicalize(&old) + current := estate() + current.RequestedScope = []inventory.ScopeEndpoint{{Role: "dpus", Endpoint: "https://192.0.2.1", TLSTrustDigest: "custom-ca-after"}} + Canonicalize(¤t) + Expect(scopeConflict(&old, ¤t)).To(BeTrue()) + Expect(inventory.IdentityConflict(&old, ¤t)).To(BeFalse()) + current.ScopeDigest = "" // Required management observation failed, so no complete scope exists. + Expect(scopeConflict(&old, ¤t)).To(BeFalse()) + } + old := estate() + old.RequestedScope = []inventory.ScopeEndpoint{{Role: "hosts", Endpoint: "https://192.0.2.2", TLSTrustDigest: "system-roots"}} + Canonicalize(&old) + current := old + Canonicalize(¤t) + Expect(scopeConflict(&old, ¤t)).To(BeFalse()) + }) + It("records the resolved TLS policy even when collection fails before remote access", func() { + Expect(os.Setenv("DPF_SCOPE_TEST_PASSWORD", "private-password")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_SCOPE_TEST_PASSWORD") + input, err := config.Decode([]byte(`management: + kubeconfigRef: {file: missing-kubeconfig} +credentials: + reader: {username: reader, passwordRef: {env: DPF_SCOPE_TEST_PASSWORD}} +discovery: + dpus: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: reader, tls: {insecureSkipVerify: true}}] + hosts: [{bmcEndpoint: 'https://192.0.2.2', credentialRef: reader}] + probes: {mode: inventory} +`), GinkgoT().TempDir()+"/site.yaml") + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(input) + Expect(err).NotTo(HaveOccurred()) + report := Collect(context.Background(), req, "test", Dependencies{}) + Expect(report.Inventory.RequestedScope).To(HaveLen(2)) + Expect(report.Inventory.RequestedScope[0].TLSInsecureSkipVerify).To(BeTrue()) + Expect(report.Inventory.RequestedScope[1].TLSInsecureSkipVerify).To(BeFalse()) + Expect(report.Errors[0].Code).To(Equal("INVALID_MANAGEMENT_ACCESS")) + }) + It("retains insecure BMC policy in saved scope and rejects a silent trust-policy change", func() { + old := estate() + old.RequestedScope = []inventory.ScopeEndpoint{{Role: "dpus", Endpoint: "https://192.0.2.1", TLSInsecureSkipVerify: false}} + Canonicalize(&old) + current := estate() + current.RequestedScope = []inventory.ScopeEndpoint{{Role: "dpus", Endpoint: "https://192.0.2.1", TLSInsecureSkipVerify: true}} + Canonicalize(¤t) + Expect(scopeConflict(&old, ¤t)).To(BeTrue()) + data, err := json.Marshal(current) + Expect(err).NotTo(HaveOccurred()) + Expect(string(data)).To(ContainSubstring(`"tlsInsecureSkipVerify":true`)) + }) + It("distinguishes infrastructure-only scope from management scope", func() { + old := estate() + Canonicalize(&old) + current := estate() + current.Management = nil + Canonicalize(¤t) + Expect(current.ScopeDigest).NotTo(BeEmpty()) + Expect(scopeConflict(&old, ¤t)).To(BeTrue()) + current = estate() + current.Management.SystemNamespaceUID = "replacement" + Canonicalize(¤t) + Expect(scopeConflict(&old, ¤t)).To(BeTrue()) + }) + It("keeps address lookup material stable after YAML removes only provenance references", func() { + before := estate() + before.AddressLookup = &inventory.AddressLookup{Candidates: []inventory.AddressCandidate{{ + Name: "dpu-a", Role: "dpus", MAC: "02:00:00:00:00:01", Address: "192.0.2.44", Source: "neighbor-cache", EvidenceRef: "transient-probe-1", + }}, Observations: map[string]inventory.Observation{ + "source.0": {State: "observed", Values: []string{"source-bytes-digest-old"}}, + "device.dpu-a": {State: "candidate", Values: []string{"192.0.2.44"}, ReasonCode: "ADDRESS_UNVERIFIED", Message: "verify candidate"}, + }} + originalLookup, err := json.Marshal(before.AddressLookup) + Expect(err).NotTo(HaveOccurred()) + Canonicalize(&before) + original, err := json.Marshal(before) + Expect(err).NotTo(HaveOccurred()) + saved, err := yaml.Marshal(before) + Expect(err).NotTo(HaveOccurred()) + var reloaded inventory.Inventory + Expect(yaml.Unmarshal(saved, &reloaded)).To(Succeed()) + Expect(reloaded.AddressLookup.Candidates[0].EvidenceRef).To(BeEmpty()) + Canonicalize(&reloaded) + Expect(reloaded.Fingerprint).To(Equal(before.Fingerprint), "%#v", Diff(&before, &reloaded)) + Expect(Diff(&before, &reloaded)).To(BeEmpty()) + after, err := json.Marshal(before) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(original), "comparison must not mutate the source inventory") + lookupAfterJSON, err := json.Marshal(before.AddressLookup) + Expect(err).NotTo(HaveOccurred()) + Expect(lookupAfterJSON).To(Equal(originalLookup), "material projection must not strip provenance from its input") + + for _, change := range []struct { + name string + edit func(*inventory.AddressCandidate) + }{ + {name: "name", edit: func(candidate *inventory.AddressCandidate) { candidate.Name = "dpu-b" }}, + {name: "role", edit: func(candidate *inventory.AddressCandidate) { candidate.Role = "hosts" }}, + {name: "address", edit: func(candidate *inventory.AddressCandidate) { candidate.Address = "192.0.2.45" }}, + {name: "MAC", edit: func(candidate *inventory.AddressCandidate) { candidate.MAC = "02:00:00:00:00:02" }}, + {name: "source", edit: func(candidate *inventory.AddressCandidate) { candidate.Source = "arp-table" }}, + } { + By(change.name) + current := reloaded + current.AddressLookup = &inventory.AddressLookup{Candidates: slices.Clone(reloaded.AddressLookup.Candidates), Observations: maps.Clone(reloaded.AddressLookup.Observations)} + Expect(Diff(&reloaded, ¤t)).To(BeEmpty(), "the candidate edit must be the only material change") + change.edit(¤t.AddressLookup.Candidates[0]) + Canonicalize(¤t) + Expect(current.Fingerprint).NotTo(Equal(reloaded.Fingerprint)) + Expect(Diff(&reloaded, ¤t)).NotTo(BeEmpty()) + } + for _, change := range []struct { + name string + edit func(*inventory.Observation) + }{ + {name: "state", edit: func(observation *inventory.Observation) { observation.State = "unavailable" }}, + {name: "reason", edit: func(observation *inventory.Observation) { observation.ReasonCode = "ADDRESS_SOURCE_UNREADABLE" }}, + } { + By(change.name) + current := reloaded + current.AddressLookup = &inventory.AddressLookup{Candidates: slices.Clone(reloaded.AddressLookup.Candidates), Observations: maps.Clone(reloaded.AddressLookup.Observations)} + observation := current.AddressLookup.Observations["source.0"] + change.edit(&observation) + current.AddressLookup.Observations["source.0"] = observation + Canonicalize(¤t) + Expect(current.Fingerprint).NotTo(Equal(reloaded.Fingerprint)) + Expect(Diff(&reloaded, ¤t)).NotTo(BeEmpty(), "source state and reason remain material") + } + current := reloaded + current.AddressLookup = &inventory.AddressLookup{Candidates: slices.Clone(reloaded.AddressLookup.Candidates), Observations: maps.Clone(reloaded.AddressLookup.Observations)} + candidate := current.AddressLookup.Candidates[0] + candidate.EvidenceRef = "different-transient-ref" + current.AddressLookup.Candidates[0] = candidate + source := current.AddressLookup.Observations["source.0"] + source.Values = []string{"source-bytes-digest-new"} + current.AddressLookup.Observations["source.0"] = source + device := current.AddressLookup.Observations["device.dpu-a"] + device.Values = []string{"192.0.2.44"} + current.AddressLookup.Observations["device.dpu-a"] = device + Canonicalize(¤t) + Expect(current.Fingerprint).To(Equal(reloaded.Fingerprint)) + Expect(Diff(&reloaded, ¤t)).To(BeEmpty()) + + current = reloaded + current.AddressLookup = &inventory.AddressLookup{Candidates: []inventory.AddressCandidate{}, Observations: maps.Clone(reloaded.AddressLookup.Observations)} + device = current.AddressLookup.Observations["device.dpu-a"] + device.State, device.Values, device.ReasonCode = "unknown", nil, "ADDRESS_LEASE_EXPIRED" + current.AddressLookup.Observations["device.dpu-a"] = device + Canonicalize(¤t) + Expect(current.Fingerprint).NotTo(Equal(reloaded.Fingerprint), "expired candidate status remains material") + Expect(Diff(&reloaded, ¤t)).NotTo(BeEmpty()) + + request, err := addresslookup.Validate(addresslookup.Config{ + Sources: []addresslookup.Source{{File: "leases", Format: "dnsmasq-leases"}}, + Devices: []addresslookup.Selector{{Name: "dpu-a", Role: "arm", MAC: "02:00:00:00:00:01"}}, + }) + Expect(err).NotTo(HaveOccurred()) + first, firstEvidence := addresslookup.Collect(request, time.Unix(100, 0), func(string, int64) ([]byte, error) { + return []byte("200 02:00:00:00:00:01 192.0.2.44 host-one *\n"), nil + }, nil) + second, secondEvidence := addresslookup.Collect(request, time.Unix(100, 0), func(string, int64) ([]byte, error) { + return []byte("200 02:00:00:00:00:01 192.0.2.44 host-two *\n"), nil + }, nil) + Expect(firstEvidence).NotTo(BeEmpty()) + Expect(secondEvidence).NotTo(BeEmpty()) + Expect(first.Candidates[0].EvidenceRef).NotTo(Equal(second.Candidates[0].EvidenceRef)) + beforeLookup, afterLookup := estate(), estate() + beforeLookup.AddressLookup, afterLookup.AddressLookup = first, second + Canonicalize(&beforeLookup) + Canonicalize(&afterLookup) + Expect(afterLookup.Fingerprint).To(Equal(beforeLookup.Fingerprint)) + Expect(Diff(&beforeLookup, &afterLookup)).To(BeEmpty()) + }) +}) diff --git a/pkg/dpf/discovery/inventory_mode_test.go b/pkg/dpf/discovery/inventory_mode_test.go new file mode 100644 index 00000000..bf5c76ad --- /dev/null +++ b/pkg/dpf/discovery/inventory_mode_test.go @@ -0,0 +1,181 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "context" + "os" + "path/filepath" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/artifacts" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +type fixtureCollector struct { + fail bool + admin bool + cancel context.CancelFunc +} + +func (f fixtureCollector) Collect(_ context.Context, t config.Target, _ time.Duration) (inventory.Device, map[string]inventory.Evidence, error) { + d := inventory.Device{ID: "host-one", RecordKey: t.InputRef, InputRef: t.InputRef, ObservationState: "observed", Qualification: "unqualified", InventoryCompleteness: "partial", EvidenceRefs: []string{"source"}} + if t.SSH != nil { + d.SSHEndpoint = t.Endpoint + } else { + d.BMCEndpoint = t.Endpoint + } + if t.Role == "hosts" { + d.Adapters = []inventory.Adapter{{Key: "card-one", IdentityState: "known", SerialNumber: "CARD-ONE", Model: "BlueField-4", EvidenceRefs: []string{"source"}}} + } else { + d.ID = "dpu-one" + d.SerialNumber = "CARD-ONE" + if t.SSH != nil { + d.SSHEndpoint = t.Endpoint + } else { + d.BMCEndpoint = t.Endpoint + } + if f.admin { + d.Observations = map[string]inventory.Observation{"bmc-administrator": {State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED", Message: "fixture verified"}} + } + } + if f.fail { + d.ObservationState = "unavailable" + d.ReasonCode = "CLEANUP_INCOMPLETE" + return d, nil, &redfish.ObservationError{Primary: &redfish.Error{Code: "ACCESS_DENIED", URI: "/redfish/v1/Systems", Status: 403, Method: "GET"}, Cleanup: &redfish.Error{Code: "ACCESS_DENIED", URI: "/redfish/v1/SessionService/Sessions/own", Status: 403, Method: "DELETE"}, SessionURI: "/redfish/v1/SessionService/Sessions/own"} + } + if f.cancel != nil { + f.cancel() + } + return d, map[string]inventory.Evidence{"source": {Kind: "fixture", Authenticated: true}}, nil +} +func inventoryRequest() config.Request { + return config.Request{Mode: "inventory", Timeout: time.Second, RequestTimeout: time.Second, MaxConcurrentBMCs: 2, Targets: []config.Target{{Role: "hosts", Endpoint: "https://host.invalid", InputRef: "/discovery/hosts/0"}, {Role: "dpus", Endpoint: "ssh://arm.invalid:22", InputRef: "/discovery/dpus/0", SSH: &config.SSHAccess{}}}} +} + +var _ = Describe("independent inventory acceptance", func() { + It("completes without Kubernetes when DPU BMC administrator privileges are verified, retaining qualification and readiness gaps", func() { + req := inventoryRequest() + req.Targets[1].SSH = nil + req.Targets[1].Endpoint = "https://dpu.invalid" + r := Collect(context.Background(), req, "run", Dependencies{BMC: fixtureCollector{admin: true}}) + Expect(ExitCode(r)).To(Equal(0)) + Expect(r.Inventory.Management).To(BeNil()) + Expect(r.Inventory.Associations).To(HaveLen(1)) + Expect(r.Acceptance).To(Equal("not-evaluated")) + Expect(r.Inventory.ScopeDigest).NotTo(BeEmpty()) + Expect(r.Discovery.Checks).To(ContainElement(And(HaveField("ID", "dpf.device.qualification"), HaveField("State", "failed"), HaveField("RequiredPhase", "generate")))) + Expect(r.Discovery.Checks).To(ContainElement(And(HaveField("ID", "dpf.management.dpf-authentication"), HaveField("State", "not-evaluated"), HaveField("RequiredPhase", "deploy")))) + }) + It("derives only host-reported card identity without inventing a BMC or ARM binding", func() { + req := inventoryRequest() + req.Targets = req.Targets[:1] + r := Collect(context.Background(), req, "run", Dependencies{BMC: fixtureCollector{}}) + Expect(ExitCode(r)).To(Equal(2)) + Expect(r.Discovery.Outcome).To(Equal("partial")) + Expect(r.Inventory.DPUs).To(HaveLen(1)) + Expect(r.Inventory.Associations).To(HaveLen(1)) + Expect(r.Inventory.DPUs[0].BMCEndpoint).To(BeEmpty()) + Expect(r.Inventory.DPUs[0].SSHEndpoint).To(BeEmpty()) + Expect(r.Inventory.DPUs[0].ReaderAdapter).To(Equal("host-card-v1")) + }) + It("fails closed for ARM SSH even when the collector claims admin or reports a local BMC address", func() { + r := Collect(context.Background(), inventoryRequest(), "run", Dependencies{BMC: fixtureCollector{}, SSH: fixtureCollector{admin: true}}) + Expect(ExitCode(r)).To(Equal(2)) + Expect(r.Discovery.Outcome).To(Equal("partial")) + Expect(r.Inventory.Associations).To(HaveLen(1)) + Expect(r.Discovery.Checks).To(ContainElement(And(HaveField("ID", "dpf.management.bmc-administrator"), HaveField("State", "failed"), HaveField("RequiredPhase", "discover")))) + }) + It("publishes completed admin-verified inventory and preserves its handoff after missing admin evidence", func() { + dir, err := filepath.EvalSymlinks(GinkgoT().TempDir()) + Expect(err).NotTo(HaveOccurred()) + seed := []byte("credentials:\n read: {username: reader, passwordRef: {env: DPF_PUBLICATION_PASSWORD}}\ndiscovery:\n hosts: [{bmcEndpoint: 'https://host.invalid', credentialRef: read}]\n dpus: [{bmcEndpoint: 'https://dpu.invalid', credentialRef: read}]\n probes: {mode: inventory}\n") + Expect(os.WriteFile(filepath.Join(dir, "seed.yaml"), seed, 0600)).To(Succeed()) + input, err := config.Load(filepath.Join(dir, "seed.yaml")) + Expect(err).NotTo(HaveOccurred()) + req := inventoryRequest() + req.Input = input + req.Targets[1].SSH = nil + req.Targets[1].Endpoint = "https://dpu.invalid" + output := filepath.Join(dir, "discovered.yaml") + for _, verified := range []bool{true, false} { + store, err := artifacts.New(input, output, true) + Expect(err).NotTo(HaveOccurred()) + report := Collect(context.Background(), req, store.RunID(), Dependencies{BMC: fixtureCollector{admin: verified}}) + Expect(store.Publish(context.Background(), &report)).To(Succeed()) + if verified { + Expect(ExitCode(report)).To(Equal(0)) + } else { + Expect(ExitCode(report)).To(Equal(2)) + Expect(report.Artifacts).To(ContainElement(HaveField("Path", filepath.Join(store.RunDir, "partial.yaml")))) + partial, err := os.ReadFile(filepath.Join(store.RunDir, "partial.yaml")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(partial)).To(ContainSubstring("BMC_ADMIN_NOT_VERIFIED")) + } + saved, err := config.Load(output) + Expect(err).NotTo(HaveOccurred()) + Expect(saved.Site.Inventory.DPUs[0].Observations["bmc-administrator"].ReasonCode).To(Equal("BMC_ADMIN_VERIFIED")) + } + }) + It("retains observation and cleanup failures instead of substituting another source", func() { + r := Collect(context.Background(), inventoryRequest(), "run", Dependencies{BMC: fixtureCollector{fail: true}, SSH: fixtureCollector{}}) + Expect(ExitCode(r)).To(Equal(3)) + Expect(r.Inventory.Associations).To(BeEmpty()) + Expect(r.Discovery.Cleanup.State).To(Equal("incomplete")) + Expect(r.Errors).To(ContainElement(And(HaveField("HTTPStatus", 403), HaveField("Method", "GET"), HaveField("Stage", "observation")))) + Expect(r.Errors).To(ContainElement(And(HaveField("HTTPStatus", 403), HaveField("Method", "DELETE"), HaveField("Stage", "cleanup")))) + Expect(r.Discovery.Cleanup.Leftovers).To(ContainElement(ContainSubstring("/Sessions/own"))) + }) + It("requires a fresh seed for legacy serial-keyed host cards without rejecting stable or explicit bindings", func() { + req := inventoryRequest() + req.Targets = req.Targets[:1] + first := Collect(context.Background(), req, "first", Dependencies{BMC: fixtureCollector{}}) + Expect(ExitCode(first)).To(Equal(2)) + Expect(inventory.LegacyHostCardBindings(first.Inventory)).To(BeFalse()) + host := &first.Inventory.Hosts[0] + host.Adapters[0].Key = inventory.Digest("dpf-adapter-v1", []string{host.ID, host.Adapters[0].SerialNumber}) + first.Inventory.DPUs = HostCards(first.Inventory.Hosts) + Expect(inventory.LegacyHostCardBindings(first.Inventory)).To(BeTrue()) + req.Input.Site.Inventory = first.Inventory + current := Collect(context.Background(), req, "second", Dependencies{BMC: fixtureCollector{}}) + Expect(current.Discovery.Outcome).To(Equal("partial")) + Expect(current.Errors).To(ContainElement(And(HaveField("Code", "SCOPE_CONFLICT"), HaveField("Message", ContainSubstring("legacy Redfish card bindings"))))) + first.Inventory.DPUs[0].ReaderAdapter = "explicit-endpoint" + Expect(inventory.LegacyHostCardBindings(first.Inventory)).To(BeFalse()) + first.Inventory.DPUs = HostCards(first.Inventory.Hosts) + host.Adapters[0].SerialNumber = "" + Expect(inventory.LegacyHostCardBindings(first.Inventory)).To(BeFalse()) + }) + It("does not leave a completed handoff after cancellation of otherwise valid observations", func() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + req := inventoryRequest() + req.Targets = req.Targets[:1] + r := Collect(ctx, req, "run", Dependencies{BMC: fixtureCollector{cancel: cancel}}) + Expect(r.Inventory.Associations).To(HaveLen(1)) + Expect(r.Discovery.Outcome).To(Equal("partial")) + Expect(r.Execution).To(Equal("cancelled")) + Expect(ExitCode(r)).To(Equal(1)) + }) + + It("keeps unknown-serial BlueFields and rejects fabricated or ambiguous associations", func() { + h := inventory.Device{ID: "host", RecordKey: "h", ObservationState: "observed", Adapters: []inventory.Adapter{{Key: "one", Model: "BlueField-4"}}} + d := HostCards([]inventory.Device{h}) + Expect(d).To(HaveLen(1)) + Expect(d[0].ID).To(BeEmpty()) + a, c := Correlate(d, []inventory.Device{h}) + Expect(a).To(BeEmpty()) + Expect(c[0].State).To(Equal("failed")) + h.Adapters[0].SerialNumber = "SAME" + h.Adapters = append(h.Adapters, inventory.Adapter{Key: "two", Model: "BlueField-4", SerialNumber: "SAME"}) + a, c = Correlate(HostCards([]inventory.Device{h}), []inventory.Device{h}) + Expect(a).To(BeEmpty()) + Expect(c[0].State).To(Equal("failed")) + }) +}) diff --git a/pkg/dpf/discovery/management.go b/pkg/dpf/discovery/management.go new file mode 100644 index 00000000..f43257ec --- /dev/null +++ b/pkg/dpf/discovery/management.go @@ -0,0 +1,241 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "context" + "crypto/sha256" + "encoding/json" + "errors" + "fmt" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "net/url" + "slices" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/rest" + "k8s.io/client-go/tools/clientcmd" +) + +type ManagementClient struct { + Client kubernetes.Interface + REST *rest.Config + Namespace string +} + +func ManagementAccess(r config.Request) (ManagementClient, error) { + rules := &clientcmd.ClientConfigLoadingRules{ExplicitPath: r.Kubeconfig} + content, err := config.ReadReference(r.Kubeconfig, 16<<20) + if err != nil { + return ManagementClient{}, fmt.Errorf("invalid bounded management kubeconfig") + } + raw, err := clientcmd.Load(content) + if err != nil { + return ManagementClient{}, fmt.Errorf("cannot load explicit management kubeconfig") + } + for _, a := range raw.AuthInfos { + a.LocationOfOrigin = r.Kubeconfig + } + for _, c := range raw.Clusters { + c.LocationOfOrigin = r.Kubeconfig + } + if err := clientcmd.ResolveLocalPaths(raw); err != nil { + return ManagementClient{}, fmt.Errorf("cannot resolve kubeconfig file references") + } + selected := r.Context + if selected == "" { + selected = raw.CurrentContext + if raw.Contexts[selected] == nil && len(raw.Contexts) == 1 { + for k := range raw.Contexts { + selected = k + } + } + } + contextConfig := raw.Contexts[selected] + if contextConfig == nil { + return ManagementClient{}, fmt.Errorf("management context selection is required") + } + auth := raw.AuthInfos[contextConfig.AuthInfo] + cluster := raw.Clusters[contextConfig.Cluster] + if auth == nil || cluster == nil { + return ManagementClient{}, fmt.Errorf("management context has missing auth/cluster") + } + if auth.Exec != nil || auth.AuthProvider != nil || auth.Username != "" || auth.Password != "" { + return ManagementClient{}, fmt.Errorf("management kubeconfig requires certificate/key or bearer token; exec/auth-provider/basic authentication unsupported") + } + if auth.Token != "" && auth.TokenFile != "" || len(auth.ClientCertificateData) > 0 && auth.ClientCertificate != "" || len(auth.ClientKeyData) > 0 && auth.ClientKey != "" || len(cluster.CertificateAuthorityData) > 0 && cluster.CertificateAuthority != "" { + return ManagementClient{}, fmt.Errorf("management auth/trust references must use exactly one file or embedded value") + } + hasToken := auth.Token != "" || auth.TokenFile != "" + hasCertificate := (auth.ClientCertificate != "" || len(auth.ClientCertificateData) > 0) && (auth.ClientKey != "" || len(auth.ClientKeyData) > 0) + if !hasToken && !hasCertificate { + return ManagementClient{}, fmt.Errorf("management certificate/key or bearer token is required") + } + u, err := url.Parse(cluster.Server) + if err != nil || u.Scheme != "https" || u.Hostname() == "" || u.User != nil || u.RawQuery != "" || u.Fragment != "" || cluster.InsecureSkipTLSVerify { + return ManagementClient{}, fmt.Errorf("management server requires verified HTTPS without userinfo/query/fragment") + } + for _, reference := range []struct { + file *string + data *[]byte + }{{&auth.ClientCertificate, &auth.ClientCertificateData}, {&auth.ClientKey, &auth.ClientKeyData}, {&cluster.CertificateAuthority, &cluster.CertificateAuthorityData}} { + if *reference.file != "" && len(*reference.data) == 0 { + b, e := config.ReadReference(*reference.file, 1<<20) + if e != nil { + return ManagementClient{}, fmt.Errorf("invalid bounded management auth/trust reference") + } + *reference.data = b + *reference.file = "" + } + } + if auth.TokenFile != "" && auth.Token == "" { + b, e := config.ReadReference(auth.TokenFile, 64<<10) + if e != nil { + return ManagementClient{}, fmt.Errorf("invalid bounded token reference") + } + auth.Token = strings.TrimSpace(string(b)) + if auth.Token == "" { + return ManagementClient{}, fmt.Errorf("empty management token reference") + } + auth.TokenFile = "" + } + loader := clientcmd.NewNonInteractiveClientConfig(*raw, selected, &clientcmd.ConfigOverrides{}, rules) + rc, err := loader.ClientConfig() + if err != nil { + return ManagementClient{}, fmt.Errorf("cannot resolve management certificate/token configuration") + } + rc.Timeout = r.RequestTimeout + rc.CAFile = "" + client, err := kubernetes.NewForConfig(rc) + if err != nil { + return ManagementClient{}, fmt.Errorf("cannot construct management client") + } + ns := r.Namespace + if ns == "" { + ns = contextConfig.Namespace + } + if ns == "" { + ns = "default" + } + return ManagementClient{Client: client, REST: rc, Namespace: ns}, nil +} +func (m ManagementClient) Observe(ctx context.Context) (*inventory.Management, error) { + ns, err := m.Client.CoreV1().Namespaces().Get(ctx, "kube-system", metav1.GetOptions{}) + if err != nil { + return nil, fmt.Errorf("management cluster identity read failed") + } + if ns.UID == "" { + return nil, fmt.Errorf("management identity UID is empty") + } + // REST request honors caller cancellation, unlike ServerVersion's context-free API. + var version struct { + GitVersion string `json:"gitVersion"` + } + data, err := m.Client.Discovery().RESTClient().Get().AbsPath("/version").Do(ctx).Raw() + if err != nil || json.Unmarshal(data, &version) != nil || version.GitVersion == "" { + return nil, fmt.Errorf("management version read failed") + } + result := &inventory.Management{APIEndpoint: m.REST.Host, SystemNamespaceUID: string(ns.UID), KubernetesVersion: version.GitVersion, Namespace: m.Namespace, TrustDigest: fmt.Sprintf("sha256:%x", sha256.Sum256(m.REST.CAData)), Nodes: []inventory.Node{}, EvidenceRefs: []string{"management-identity", "management-nodes"}} + if len(m.REST.CAData) == 0 { + result.TrustDigest = "system-roots" + } + if m.REST.ServerName != "" { + result.TrustDigest = inventory.Digest("management-trust-v1", []string{result.TrustDigest, m.REST.ServerName}) + } + cursor := "" + for { + nodes, err := m.Client.CoreV1().Nodes().List(ctx, metav1.ListOptions{Limit: 64, Continue: cursor}) + if err != nil { + return nil, fmt.Errorf("management node inspection failed") + } + for _, n := range nodes.Items { + node := inventory.Node{Name: n.Name, UID: string(n.UID), Architecture: n.Status.NodeInfo.Architecture, OS: n.Status.NodeInfo.OperatingSystem, Unschedulable: n.Spec.Unschedulable, Addresses: []string{}} + _, cp := n.Labels["node-role.kubernetes.io/control-plane"] + _, master := n.Labels["node-role.kubernetes.io/master"] + node.ControlPlane = cp || master + for _, condition := range n.Status.Conditions { + if condition.Type == corev1.NodeReady { + node.Ready = condition.Status == corev1.ConditionTrue + } + } + for _, address := range n.Status.Addresses { + node.Addresses = append(node.Addresses, address.Address) + } + slices.Sort(node.Addresses) + result.Nodes = append(result.Nodes, node) + } + if len(result.Nodes) > 64 { + return nil, fmt.Errorf("management node scope exceeds 64 nodes") + } + cursor = nodes.Continue + if cursor == "" { + break + } + } + result.Observations = map[string]inventory.Observation{} + storage, storageErr := m.Client.StorageV1().StorageClasses().List(ctx, metav1.ListOptions{Limit: 256}) + storageFact := apiObservationFailure(storageErr) + if storageErr == nil && storage.Continue != "" { + storageFact = inventory.Observation{State: "unknown", ReasonCode: "API_COVERAGE_INCOMPLETE", Message: "StorageClass enumeration exceeded the bounded first page; not all entries were observed."} + } + if storageErr == nil && storage.Continue == "" { + storageFact.State = "known" + storageFact.ReasonCode = "" + storageFact.Message = "" + for _, s := range storage.Items { + storageFact.Values = append(storageFact.Values, s.Name+"/"+s.Provisioner) + } + slices.Sort(storageFact.Values) + } + result.Observations["storageClasses"] = storageFact + data, groupErr := m.Client.Discovery().RESTClient().Get().AbsPath("/apis").Do(ctx).Raw() + var groups metav1.APIGroupList + groupsFact := apiObservationFailure(groupErr) + decodeErr := json.Unmarshal(data, &groups) + if groupErr == nil && decodeErr != nil { + groupsFact = inventory.Observation{State: "unknown", ReasonCode: "API_RESPONSE_INVALID", Message: "The API group discovery response is not valid APIGroupList JSON."} + } + if groupErr == nil && decodeErr == nil { + groupsFact.State = "known" + groupsFact.ReasonCode = "" + groupsFact.Message = "" + for _, g := range groups.Groups { + if strings.HasSuffix(g.Name, "dpu.nvidia.com") { + groupsFact.Values = append(groupsFact.Values, g.Name) + } + } + slices.Sort(groupsFact.Values) + } + result.Observations["advertisedDPFAPIGroups"] = groupsFact + slices.SortFunc(result.Nodes, func(a, b inventory.Node) int { return strings.Compare(a.UID, b.UID) }) + return result, nil +} + +// NewEvidence binds API reads to the current collection rather than old success flags. +func NewEvidence(at time.Time) map[string]inventory.Evidence { + return map[string]inventory.Evidence{"management-identity": {Kind: "kubernetes-api", InputRef: "/management", ResourceURI: "/api/v1/namespaces/kube-system", Property: "metadata.uid", ObservedAt: at, Vantage: "executor", Authenticated: true}, "management-nodes": {Kind: "kubernetes-api", InputRef: "/management", ResourceURI: "/api/v1/nodes", ObservedAt: at, Vantage: "executor", Authenticated: true}} +} + +// apiObservationFailure preserves why optional API evidence is unavailable without raw errors. +func apiObservationFailure(err error) inventory.Observation { + code, message := "API_INSPECTION_FAILED", "The Kubernetes API request failed; no complete observation was obtained." + switch { + case apierrors.IsForbidden(err): + code, message = "API_ACCESS_DENIED", "The configured Kubernetes identity is forbidden from reading this resource." + case apierrors.IsUnauthorized(err): + code, message = "API_AUTHENTICATION_FAILED", "The Kubernetes API did not accept the configured credentials." + case apierrors.IsNotFound(err): + code, message = "API_RESOURCE_NOT_FOUND", "The requested Kubernetes API resource is not available." + case errors.Is(err, context.Canceled): + code, message = "API_CANCELLED", "The Kubernetes API inspection was cancelled." + case errors.Is(err, context.DeadlineExceeded), apierrors.IsTimeout(err): + code, message = "API_TIMEOUT", "The Kubernetes API inspection timed out." + } + return inventory.Observation{State: "unknown", Values: []string{}, ReasonCode: code, Message: message} +} diff --git a/pkg/dpf/discovery/management_test.go b/pkg/dpf/discovery/management_test.go new file mode 100644 index 00000000..3580a973 --- /dev/null +++ b/pkg/dpf/discovery/management_test.go @@ -0,0 +1,128 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "context" + "encoding/json" + "encoding/pem" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/client-go/tools/clientcmd" + clientapi "k8s.io/client-go/tools/clientcmd/api" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sync/atomic" + "time" +) + +var _ = Describe("explicit management access", func() { + It("freezes CA files and preserves a proxy base path while observing three nodes", func() { + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + Expect(r.Header.Get("Authorization")).To(Equal("Bearer test-token")) + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/prefix/version": + json.NewEncoder(w).Encode(map[string]any{"gitVersion": "v1.34.0"}) + case "/prefix/api/v1/namespaces/kube-system": + w.Write([]byte(`{"apiVersion":"v1","kind":"Namespace","metadata":{"name":"kube-system","uid":"cluster-one"}}`)) + case "/prefix/api/v1/nodes": + w.Write([]byte(`{"apiVersion":"v1","kind":"NodeList","metadata":{},"items":[{"metadata":{"name":"cp1","uid":"node1","labels":{"node-role.kubernetes.io/control-plane":""}},"status":{"nodeInfo":{"architecture":"amd64","operatingSystem":"linux"},"conditions":[{"type":"Ready","status":"True"}]}}]}`)) + default: + w.WriteHeader(404) + } + })) + defer server.Close() + dir, e := os.MkdirTemp("", "dpf-management-") + Expect(e).NotTo(HaveOccurred()) + DeferCleanup(os.RemoveAll, dir) + ca := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw}) + caPath := filepath.Join(dir, "ca.pem") + Expect(os.WriteFile(caPath, ca, 0600)).To(Succeed()) + raw := clientapi.Config{CurrentContext: "management", Clusters: map[string]*clientapi.Cluster{"management": {Server: server.URL + "/prefix", CertificateAuthority: caPath}}, AuthInfos: map[string]*clientapi.AuthInfo{"reader": {Token: "test-token"}}, Contexts: map[string]*clientapi.Context{"management": {Cluster: "management", AuthInfo: "reader"}}} + b, e := clientcmd.Write(raw) + Expect(e).NotTo(HaveOccurred()) + filename := filepath.Join(dir, "config") + Expect(os.WriteFile(filename, b, 0600)).To(Succeed()) + mg, e := ManagementAccess(config.Request{Kubeconfig: filename, RequestTimeout: time.Second}) + Expect(e).NotTo(HaveOccurred()) + Expect(mg.REST.CAData).To(Equal(ca)) + Expect(mg.REST.CAFile).To(BeEmpty()) + observed, e := mg.Observe(context.Background()) + Expect(e).NotTo(HaveOccurred()) + Expect(observed.SystemNamespaceUID).To(Equal("cluster-one")) + Expect(observed.TrustDigest).NotTo(Equal("system-roots")) + Expect(observed.Nodes).To(HaveLen(1)) + }) + It("rejects anonymous, exec and HTTP configurations before client construction", func() { + dir, e := os.MkdirTemp("", "dpf-management-") + Expect(e).NotTo(HaveOccurred()) + DeferCleanup(os.RemoveAll, dir) + for _, test := range []struct { + server string + auth *clientapi.AuthInfo + }{{"https://example.invalid", &clientapi.AuthInfo{}}, {"http://example.invalid", &clientapi.AuthInfo{Token: "private-token"}}, {"https://example.invalid", &clientapi.AuthInfo{Exec: &clientapi.ExecConfig{Command: "must-never-execute"}}}} { + raw := clientapi.Config{CurrentContext: "m", Clusters: map[string]*clientapi.Cluster{"m": {Server: test.server}}, AuthInfos: map[string]*clientapi.AuthInfo{"r": test.auth}, Contexts: map[string]*clientapi.Context{"m": {Cluster: "m", AuthInfo: "r"}}} + b, e := clientcmd.Write(raw) + Expect(e).NotTo(HaveOccurred()) + filename := filepath.Join(dir, "config") + Expect(os.WriteFile(filename, b, 0600)).To(Succeed()) + _, e = ManagementAccess(config.Request{Kubeconfig: filename}) + Expect(e).To(HaveOccurred()) + Expect(e.Error()).NotTo(ContainSubstring("private-token")) + } + }) +}) + +var _ = Describe("management observation failures", func() { + It("distinguishes unavailable initial/recheck observations from changed identities", func() { + for _, initialFailure := range []bool{true, false} { + var reads atomic.Int32 + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/api/v1/namespaces/kube-system": + if reads.Add(1) > 1 || initialFailure { + w.WriteHeader(403) + return + } + w.Write([]byte(`{"apiVersion":"v1","kind":"Namespace","metadata":{"name":"kube-system","uid":"cluster-one"}}`)) + case "/version": + w.Write([]byte(`{"gitVersion":"v1.34.0"}`)) + case "/api/v1/nodes": + w.Write([]byte(`{"apiVersion":"v1","kind":"NodeList","items":[]}`)) + default: + w.WriteHeader(404) + } + })) + raw := clientapi.Config{CurrentContext: "m", Clusters: map[string]*clientapi.Cluster{"m": {Server: server.URL, CertificateAuthorityData: pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw})}}, AuthInfos: map[string]*clientapi.AuthInfo{"r": {Token: "test-token"}}, Contexts: map[string]*clientapi.Context{"m": {Cluster: "m", AuthInfo: "r"}}} + data, err := clientcmd.Write(raw) + Expect(err).NotTo(HaveOccurred()) + filename := filepath.Join(GinkgoT().TempDir(), "management.kubeconfig") + Expect(os.WriteFile(filename, data, 0600)).To(Succeed()) + input := config.Input{} + if initialFailure { + input.Site.Inventory = &inventory.Inventory{ScopeDigest: "known-baseline"} + } + report := Collect(context.Background(), config.Request{Input: input, Kubeconfig: filename, Timeout: 10 * time.Second, RequestTimeout: time.Second, MaxConcurrentBMCs: 1, Mode: "inventory"}, "test", Dependencies{}) + server.Close() + codes := []string{} + for _, problem := range report.Errors { + codes = append(codes, problem.Code) + } + if initialFailure { + Expect(codes).To(ContainElement("MANAGEMENT_INSPECTION_FAILED")) + } else { + Expect(codes).To(ContainElement("MANAGEMENT_RECHECK_FAILED")) + } + Expect(codes).NotTo(ContainElement("SCOPE_CONFLICT")) + Expect(codes).NotTo(ContainElement("MANAGEMENT_SCOPE_CHANGED")) + Expect(report.Discovery.Outcome).To(Equal("partial")) + Expect(ExitCode(report)).To(Equal(3)) + } + }) +}) diff --git a/pkg/dpf/discovery/material_roundtrip_test.go b/pkg/dpf/discovery/material_roundtrip_test.go new file mode 100644 index 00000000..f15523e0 --- /dev/null +++ b/pkg/dpf/discovery/material_roundtrip_test.go @@ -0,0 +1,53 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "encoding/json" + "fmt" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +var _ = Describe("persisted management fact comparison", func() { + It("keeps management facts stable across supported codecs without mutating inputs", func() { + for _, version := range []string{"v1alpha1", "v1alpha2", inventory.FormatVersion} { + for _, codec := range []struct { + name string + encode func(any) ([]byte, error) + decode func([]byte, any) error + }{{name: "YAML", encode: yaml.Marshal, decode: yaml.Unmarshal}, {name: "JSON", encode: json.Marshal, decode: json.Unmarshal}} { + for _, node := range []inventory.Node{ + {Name: "cp-a", UID: "node-a", Addresses: nil, Interfaces: nil, Routes: nil}, + {Name: "cp-a", UID: "node-a", Addresses: []string{}, Interfaces: []inventory.Interface{}, Routes: []inventory.Route{}}, + {Name: "cp-a", UID: "node-a", Addresses: []string{"192.0.2.10"}, Interfaces: []inventory.Interface{{Name: "data0", Index: 2, MTU: 9000, Addresses: nil}}}, + {Name: "cp-a", UID: "node-a", Addresses: []string{"192.0.2.10"}, Interfaces: []inventory.Interface{{Name: "data0", Index: 2, MTU: 9000, Addresses: []string{}}}}, + } { + By(fmt.Sprintf("%s %s addresses=%#v interfaces=%#v", version, codec.name, node.Addresses, node.Interfaces)) + source := inventory.Inventory{FormatVersion: version, Management: &inventory.Management{APIEndpoint: "https://api.example.test", SystemNamespaceUID: "cluster-a", Nodes: []inventory.Node{node}}} + originalNode, err := json.Marshal(source.Management.Nodes) + Expect(err).NotTo(HaveOccurred()) + Canonicalize(&source) + encoded, err := codec.encode(source) + Expect(err).NotTo(HaveOccurred()) + var restored inventory.Inventory + Expect(codec.decode(encoded, &restored)).To(Succeed()) + Canonicalize(&restored) + Expect(restored.Fingerprint).To(Equal(source.Fingerprint)) + Expect(Diff(&source, &restored)).To(BeEmpty()) + afterNode, err := json.Marshal(source.Management.Nodes) + Expect(err).NotTo(HaveOccurred()) + Expect(afterNode).To(Equal(originalNode), "material normalization must not mutate source nodes") + restored.Management.Nodes[0].Addresses = []string{"198.51.100.10"} + Canonicalize(&restored) + Expect(restored.Fingerprint).NotTo(Equal(source.Fingerprint)) + Expect(Diff(&source, &restored)).NotTo(BeEmpty(), "a real node address change must remain visible") + } + } + } + }) +}) diff --git a/pkg/dpf/discovery/observations.go b/pkg/dpf/discovery/observations.go new file mode 100644 index 00000000..920403f1 --- /dev/null +++ b/pkg/dpf/discovery/observations.go @@ -0,0 +1,125 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "cmp" + "slices" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" +) + +// HostCards preserves host-reported BlueField identity when no independent DPU +// endpoint was requested. It does not manufacture ARM or BMC endpoint bindings. +func HostCards(hosts []inventory.Device) []inventory.Device { + devices := []inventory.Device{} + for _, host := range hosts { + if host.ObservationState != "observed" { + continue + } + for _, card := range host.Adapters { + if !strings.Contains(strings.ToLower(card.Model), "bluefield") { + continue + } + d := inventory.Device{ + RecordKey: inventory.Digest("dpf-host-card-v1", []string{host.ID, card.Key}), + InputRef: host.InputRef, Model: card.Model, SerialNumber: card.SerialNumber, + SerialNumberRaw: card.SerialNumberRaw, ReaderAdapter: "host-card-v1", + ObservationState: "observed", Qualification: "unqualified", InventoryCompleteness: "partial", + Authentication: inventory.Authentication{PasswordRead: "not-evaluated", ClientCertificateSupport: "unknown", DPFClientAuthentication: "not-evaluated"}, + BMCAddressState: "unknown", EvidenceRefs: slices.Clone(card.EvidenceRefs), + } + if redfish.ValidSerial(card.SerialNumber) { + d.IdentityKey = []string{"nvidia", card.SerialNumber} + d.ID = inventory.PhysicalID("dpu", d.IdentityKey) + } else { + d.ReasonCode = "CARD_SERIAL_MISSING" + } + devices = append(devices, d) + } + } + return devices +} + +func canonicalRuntime(r inventory.RuntimeObservation) inventory.RuntimeObservation { + r.Interfaces = canonicalEmptySlice(r.Interfaces) + for i := range r.Interfaces { + r.Interfaces[i].Addresses = canonicalEmptySlice(r.Interfaces[i].Addresses) + slices.Sort(r.Interfaces[i].Addresses) + } + slices.SortFunc(r.Interfaces, func(a, b inventory.Interface) int { + return cmp.Or(cmp.Compare(a.Index, b.Index), strings.Compare(a.Name, b.Name)) + }) + r.Routes = canonicalEmptySlice(r.Routes) + slices.SortFunc(r.Routes, func(a, b inventory.Route) int { + return cmp.Or(cmp.Compare(a.Table, b.Table), strings.Compare(a.Destination, b.Destination), strings.Compare(a.Gateway, b.Gateway), cmp.Compare(a.InterfaceIndex, b.InterfaceIndex)) + }) + r.DNS = canonicalEmptySlice(r.DNS) + slices.Sort(r.DNS) + if r.Network != nil { + n := *r.Network + n.Links = canonicalEmptySlice(n.Links) + slices.SortFunc(n.Links, func(a, b inventory.NetworkLink) int { return strings.Compare(a.Name, b.Name) }) + n.Resolver.Search = canonicalEmptySlice(n.Resolver.Search) + n.Resolver.PerLink = canonicalStringSliceMap(n.Resolver.PerLink) + n.Resolver.Domains = canonicalStringSliceMap(n.Resolver.Domains) + n.Resolver.DNSMasqServers = canonicalEmptySlice(n.Resolver.DNSMasqServers) + n.Neighbors = canonicalEmptySlice(n.Neighbors) + for i := range n.Neighbors { + n.Neighbors[i].State = canonicalEmptySlice(n.Neighbors[i].State) + slices.Sort(n.Neighbors[i].State) + } + slices.SortFunc(n.Neighbors, func(a, b inventory.Neighbor) int { + return cmp.Or(strings.Compare(a.Interface, b.Interface), strings.Compare(a.Address, b.Address), strings.Compare(a.MAC, b.MAC), slices.Compare(a.State, b.State)) + }) + n.LLDP = canonicalEmptySlice(n.LLDP) + slices.SortFunc(n.LLDP, func(a, b inventory.LLDPNeighbor) int { + return cmp.Or(strings.Compare(a.Interface, b.Interface), strings.Compare(a.Chassis, b.Chassis), strings.Compare(a.Port, b.Port)) + }) + r.Network = &n + } + if r.Hardware != nil { + h := *r.Hardware + h.Cards = canonicalEmptySlice(h.Cards) + slices.SortFunc(h.Cards, func(a, b inventory.CardHardware) int { + return cmp.Or(strings.Compare(a.Domain, b.Domain), strings.Compare(a.PCIAddress, b.PCIAddress)) + }) + r.Hardware = &h + } + return r +} + +func canonicalFunctions(functions []inventory.PCIFunction) []inventory.PCIFunction { + out := canonicalEmptySlice(functions) + for i := range out { + out[i].Netdevs = canonicalEmptySlice(out[i].Netdevs) + slices.Sort(out[i].Netdevs) + } + slices.SortFunc(out, func(a, b inventory.PCIFunction) int { return strings.Compare(a.Address, b.Address) }) + return out +} + +// canonicalEmptySlice clones a material slice and gives nil and empty slices +// one representation. The v3 YAML codec emits empty non-omitempty collections +// as [] on save, so retaining the in-memory distinction would create a false +// material change after reload. +func canonicalEmptySlice[T any](values []T) []T { + if len(values) == 0 { + return nil + } + return slices.Clone(values) +} + +func canonicalStringSliceMap(values map[string][]string) map[string][]string { + if len(values) == 0 { + return nil + } + out := make(map[string][]string, len(values)) + for key, value := range values { + out[key] = canonicalEmptySlice(value) + } + return out +} diff --git a/pkg/dpf/discovery/progress_test.go b/pkg/dpf/discovery/progress_test.go new file mode 100644 index 00000000..8a7efa31 --- /dev/null +++ b/pkg/dpf/discovery/progress_test.go @@ -0,0 +1,97 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "bytes" + "context" + "encoding/json" + "strings" + "time" + + "github.com/go-logr/zapr" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "go.uber.org/zap" + "go.uber.org/zap/zapcore" + "go.uber.org/zap/zaptest/observer" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +type progressCollector struct { + entered chan string + release chan struct{} + fail bool +} + +func (c progressCollector) Collect(ctx context.Context, target config.Target, timeout time.Duration) (inventory.Device, map[string]inventory.Evidence, error) { + ui.FromContext(ctx).Info("[RUNNING] %s: fixture observation", target.DisplayName()) + c.entered <- target.Role + select { + case <-c.release: + case <-ctx.Done(): + } + return (fixtureCollector{admin: true, fail: c.fail}).Collect(ctx, target, timeout) +} + +var _ = Describe("discovery progress and diagnostic context", func() { + It("streams concurrent named sources, respects quiet output and keeps logs secret-safe", func() { + for _, tc := range []struct{ quiet, fail bool }{{false, false}, {false, true}, {true, false}} { + var progress bytes.Buffer + core, records := observer.New(zapcore.DebugLevel) + ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second) + ctx = log.IntoContext(ctx, zapr.NewLogger(zap.New(core))) + ctx = ui.WithOutput(ctx, ui.NewWithWriter(&progress)) + if tc.quiet { + ctx = ui.WithOutput(ctx, ui.NewSilent()) + } + req := inventoryRequest() + req.Timeout = 3 * time.Second + req.Targets[0].Name = "Host A" + req.Targets[0].Endpoint = "https://url-user-canary:url-pass-canary@host.invalid/private-path-canary?token=query-canary" + req.Targets[1].Name, req.Targets[1].Endpoint, req.Targets[1].SSH = "DPU A", "https://dpu.invalid", nil + collector := progressCollector{entered: make(chan string, 2), release: make(chan struct{}), fail: tc.fail} + result := make(chan inventory.Report, 1) + go func() { result <- Collect(ctx, req, "private-run-id-canary", Dependencies{BMC: collector}) }() + Eventually(collector.entered).Should(Receive()) + Eventually(collector.entered).Should(Receive()) + // Both goroutines are blocked; these lines cannot be a post-run replay. + if !tc.quiet { + Expect(progress.String()).To(ContainSubstring(`[RUNNING] "Host A" (https://host.invalid): fixture observation`)) + Expect(progress.String()).To(ContainSubstring(`[RUNNING] "DPU A" (https://dpu.invalid): fixture observation`)) + Expect(progress.String()).NotTo(ContainSubstring("Required discovery checks completed")) + } + close(collector.release) + var report inventory.Report + Eventually(result).Should(Receive(&report)) + cancel() + Expect(records.FilterMessage("Endpoint observation started").Len()).To(Equal(2)) + Expect(records.FilterMessage("Endpoint observation completed").Len()).To(Equal(2)) + Expect(records.FilterMessage("Discovery check evaluated").Len()).To(BeNumerically(">", 0)) + Expect(records.FilterMessage("DPF discovery completed").Len()).To(Equal(1)) + if tc.quiet { + Expect(progress.String()).To(BeEmpty()) + } else { + for _, line := range strings.Split(strings.TrimSpace(progress.String()), "\n") { + Expect(line).To(MatchRegexp(`^\[(RUNNING|PASS|FAIL|UNAVAILABLE|SKIP)\] `)) + } + Expect(progress.String()).NotTo(ContainSubstring("QUALIFIED_ADAPTER_REQUIRED")) + } + if tc.fail { + Expect(report.Discovery.Outcome).To(Equal("partial")) + Expect(progress.String()).To(ContainSubstring("[FAIL] Required discovery checks")) + } else { + Expect(report.Discovery.Outcome).To(Equal("complete")) + } + encoded, err := json.Marshal(records.All()) + Expect(err).NotTo(HaveOccurred()) + for _, secret := range []string{"url-user-canary", "url-pass-canary", "private-path-canary", "query-canary", "private-run-id-canary"} { + Expect(progress.String() + string(encoded)).NotTo(ContainSubstring(secret)) + } + } + }) +}) diff --git a/pkg/dpf/discovery/reasons.go b/pkg/dpf/discovery/reasons.go new file mode 100644 index 00000000..7757ec29 --- /dev/null +++ b/pkg/dpf/discovery/reasons.go @@ -0,0 +1,29 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + +// ExplainCheck completes report diagnostics at the orchestration boundary. Collectors +// retain more specific failure messages; this table describes acceptance predicates. +func ExplainCheck(c *inventory.Check) { + if c == nil || c.State == "passed" || c.Message != "" { + return + } + messages := map[string]string{ + "EXACTLY_THREE_CONTROL_PLANES": "The management topology does not contain exactly three control-plane nodes.", + "PHYSICAL_IDENTITY_REQUIRED": "The selected source did not establish a usable physical identity; inspect its observation and error reasons.", + "QUALIFIED_ADAPTER_REQUIRED": "This hardware/firmware and reader combination has not passed release qualification; inventory success does not establish provisioning support.", + "DPU_IDENTITY_REQUIRED": "No DPU identity was established from the selected sources.", + "UNIQUE_PHYSICAL_MATCH_REQUIRED": "The DPU serial must match exactly one observed host adapter; missing, duplicate or conflicting identity prevents association.", + "HOST_ADAPTER_VENDOR_CONFLICT": "A host adapter with the DPU serial reports a non-NVIDIA PCI vendor. Verify the selected endpoints and the adapter serial/PCI inventory, then rerun discovery; a matching serial cannot override contradictory vendor identity.", + "REDFISH_AUTHORIZATION_NOT_EVALUATED": "This run did not verify permission to read DPU Redfish inventory; SSH access is a separate authorization path.", + "DPF_AUTHENTICATION_NOT_EVALUATED": "The installed DPF controller authentication path is checked during provisioning, not initial inventory.", + "INVENTORY_MODE": "Inventory mode omits management-network probes; later management readiness remains unevaluated.", + "EXACT_NETWORK_COVERAGE_REQUIRED": "Required management node, network-mode or destination evidence is missing or did not pass; inspect individual probe results.", + } + c.Message = messages[c.ReasonCode] + if c.Message == "" { + c.Message = "The check did not produce passing evidence; inspect the associated source observation and reason code." + } +} diff --git a/pkg/dpf/discovery/reasons_test.go b/pkg/dpf/discovery/reasons_test.go new file mode 100644 index 00000000..b5c48c47 --- /dev/null +++ b/pkg/dpf/discovery/reasons_test.go @@ -0,0 +1,27 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "context" + "fmt" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/runtime/schema" +) + +var _ = Describe("API diagnostic persistence", func() { + It("distinguishes authorization, absent resources, timeouts and cancellation without raw errors", func() { + resource := schema.GroupResource{Group: "storage.k8s.io", Resource: "storageclasses"} + for _, tc := range []struct { + err error + code string + }{{apierrors.NewForbidden(resource, "x", fmt.Errorf("credential-canary")), "API_ACCESS_DENIED"}, {apierrors.NewUnauthorized("credential-canary"), "API_AUTHENTICATION_FAILED"}, {apierrors.NewNotFound(resource, "x"), "API_RESOURCE_NOT_FOUND"}, {context.DeadlineExceeded, "API_TIMEOUT"}, {context.Canceled, "API_CANCELLED"}, {fmt.Errorf("credential-canary"), "API_INSPECTION_FAILED"}} { + o := apiObservationFailure(tc.err) + Expect(o.ReasonCode).To(Equal(tc.code)) + Expect(o.Message).NotTo(BeEmpty()) + Expect(o.Message).NotTo(ContainSubstring("credential-canary")) + } + }) +}) diff --git a/pkg/dpf/discovery/references_unix_test.go b/pkg/dpf/discovery/references_unix_test.go new file mode 100644 index 00000000..bc7b6be9 --- /dev/null +++ b/pkg/dpf/discovery/references_unix_test.go @@ -0,0 +1,48 @@ +//go:build !windows + +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/client-go/tools/clientcmd" + clientapi "k8s.io/client-go/tools/clientcmd/api" + "os" + "path/filepath" + "syscall" + "time" +) + +var _ = Describe("bounded reference files", func() { + It("rejects a kubeconfig FIFO before a blocking read", func() { + dir, e := os.MkdirTemp("", "dpf-fifo-") + Expect(e).NotTo(HaveOccurred()) + DeferCleanup(os.RemoveAll, dir) + filename := filepath.Join(dir, "pipe") + Expect(syscall.Mkfifo(filename, 0600)).To(Succeed()) + done := make(chan error, 1) + go func() { _, err := ManagementAccess(config.Request{Kubeconfig: filename}); done <- err }() + Eventually(done, time.Second).Should(Receive(HaveOccurred())) + }) +}) + +var _ = Describe("unambiguous credential references", func() { + It("rejects an embedded token plus FIFO tokenFile before any read", func() { + dir, e := os.MkdirTemp("", "dpf-token-") + Expect(e).NotTo(HaveOccurred()) + DeferCleanup(os.RemoveAll, dir) + fifo := filepath.Join(dir, "token") + Expect(syscall.Mkfifo(fifo, 0600)).To(Succeed()) + raw := clientapi.Config{CurrentContext: "m", Clusters: map[string]*clientapi.Cluster{"m": {Server: "https://example.invalid"}}, AuthInfos: map[string]*clientapi.AuthInfo{"a": {Token: "inline-token", TokenFile: fifo}}, Contexts: map[string]*clientapi.Context{"m": {Cluster: "m", AuthInfo: "a"}}} + b, e := clientcmd.Write(raw) + Expect(e).NotTo(HaveOccurred()) + filename := filepath.Join(dir, "config") + Expect(os.WriteFile(filename, b, 0600)).To(Succeed()) + done := make(chan error, 1) + go func() { _, err := ManagementAccess(config.Request{Kubeconfig: filename}); done <- err }() + Eventually(done, time.Second).Should(Receive(HaveOccurred())) + }) +}) diff --git a/pkg/dpf/discovery/runtime_roundtrip_test.go b/pkg/dpf/discovery/runtime_roundtrip_test.go new file mode 100644 index 00000000..42bc98b6 --- /dev/null +++ b/pkg/dpf/discovery/runtime_roundtrip_test.go @@ -0,0 +1,211 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "encoding/json" + "slices" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +var _ = Describe("runtime material round trips", func() { + It("treats empty runtime collections consistently after v3 YAML persistence", func() { + fresh := runtimeMaterialInventory(false) + explicitEmpty := runtimeMaterialInventory(true) + beforeRuntime, err := json.Marshal(fresh.DPUs[0].Runtime) + Expect(err).NotTo(HaveOccurred()) + beforeDevice, err := json.Marshal(fresh.DPUs[0]) + Expect(err).NotTo(HaveOccurred()) + beforeHostRuntime, err := json.Marshal(fresh.Hosts[0].Runtime) + Expect(err).NotTo(HaveOccurred()) + resetTypesNil := fresh.DPUs[0].AdvertisedResetTypes == nil + functionsNil := fresh.DPUs[0].Adapters[0].PCIFunctions == nil + + Canonicalize(&fresh) + Canonicalize(&explicitEmpty) + Expect(explicitEmpty.Fingerprint).To(Equal(fresh.Fingerprint)) + Expect(Diff(&fresh, &explicitEmpty)).To(BeEmpty()) + Expect(fresh.DPUs[0].Runtime.Network.Resolver.Domains).To(BeNil()) + + saved, err := yaml.Marshal(fresh) + Expect(err).NotTo(HaveOccurred()) + Expect(string(saved)).To(ContainSubstring("links: []")) + Expect(string(saved)).To(ContainSubstring("search: []")) + Expect(string(saved)).To(ContainSubstring("perLink: {}")) + Expect(string(saved)).To(ContainSubstring("domains: {}")) + Expect(string(saved)).To(ContainSubstring("neighbors: []")) + Expect(string(saved)).To(ContainSubstring("lldp: []")) + + var reloaded inventory.Inventory + Expect(yaml.Unmarshal(saved, &reloaded)).To(Succeed()) + Expect(reloaded.DPUs).To(HaveLen(1)) + Expect(reloaded.Hosts).To(HaveLen(1)) + Expect(reloaded.DPUs[0].Runtime.Network.Links).NotTo(BeNil(), "the wire representation contains an explicit empty sequence") + Expect(reloaded.DPUs[0].Runtime.Network.Resolver.Search).NotTo(BeNil()) + Expect(reloaded.DPUs[0].Runtime.Network.Resolver.PerLink).NotTo(BeNil(), "the wire representation contains an explicit empty map") + Expect(reloaded.DPUs[0].Runtime.Network.Neighbors).NotTo(BeNil()) + Expect(reloaded.DPUs[0].Runtime.Network.LLDP).NotTo(BeNil()) + Expect(reloaded.DPUs[0].Runtime.Network.Resolver.Domains).NotTo(BeNil()) + Expect(reloaded.Hosts[0].Runtime.Network.Links).NotTo(BeNil()) + + Canonicalize(&reloaded) + Expect(reloaded.Fingerprint).To(Equal(fresh.Fingerprint)) + Expect(Diff(&fresh, &reloaded)).To(BeEmpty(), "%#v", Diff(&fresh, &reloaded)) + Expect(reloaded.DPUs[0].Runtime.Observations["resolver-dns"].ReasonCode).To(Equal("TOOL_NOT_AVAILABLE"), "failed optional-parser diagnostics remain material") + Expect(reloaded.Hosts[0].Runtime.Observations["resolver-dns"].ReasonCode).To(Equal("TOOL_NOT_AVAILABLE")) + + afterRuntime, err := json.Marshal(fresh.DPUs[0].Runtime) + Expect(err).NotTo(HaveOccurred()) + Expect(afterRuntime).To(Equal(beforeRuntime), "material canonicalization must not mutate runtime observations") + afterDevice, err := json.Marshal(fresh.DPUs[0]) + Expect(err).NotTo(HaveOccurred()) + Expect(afterDevice).To(Equal(beforeDevice), "material canonicalization must not strip source facts") + afterHostRuntime, err := json.Marshal(fresh.Hosts[0].Runtime) + Expect(err).NotTo(HaveOccurred()) + Expect(afterHostRuntime).To(Equal(beforeHostRuntime), "material canonicalization must not mutate host observations") + Expect((fresh.DPUs[0].AdvertisedResetTypes == nil)).To(Equal(resetTypesNil)) + Expect((fresh.DPUs[0].Adapters[0].PCIFunctions == nil)).To(Equal(functionsNil)) + + changed := reloaded + changed.DPUs = append([]inventory.Device(nil), reloaded.DPUs...) + runtime := *reloaded.DPUs[0].Runtime + runtime.Interfaces = slices.Clone(runtime.Interfaces) + for i := range runtime.Interfaces { + runtime.Interfaces[i].Addresses = slices.Clone(runtime.Interfaces[i].Addresses) + } + changed.DPUs[0].Runtime = &runtime + changed.DPUs[0].Runtime.Interfaces[0].Addresses = []string{"192.0.2.45"} + Canonicalize(&changed) + Expect(changed.Fingerprint).NotTo(Equal(reloaded.Fingerprint), "a real host address change remains material") + Expect(Diff(&reloaded, &changed)).NotTo(BeEmpty()) + }) + + It("normalizes omitted top-level runtime slices without losing source immutability", func() { + missing := runtimeMaterialInventory(false) + empty := runtimeMaterialInventory(false) + missing.DPUs[0].Runtime.Interfaces = nil + empty.DPUs[0].Runtime.Interfaces = []inventory.Interface{} + before, err := json.Marshal(missing.DPUs[0].Runtime) + Expect(err).NotTo(HaveOccurred()) + Canonicalize(&missing) + Canonicalize(&empty) + Expect(empty.Fingerprint).To(Equal(missing.Fingerprint)) + Expect(Diff(&missing, &empty)).To(BeEmpty()) + after, err := json.Marshal(missing.DPUs[0].Runtime) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(before)) + }) + + It("compares empty collections consistently after v3 JSON persistence", func() { + fresh := runtimeMaterialInventory(false) + Canonicalize(&fresh) + empty := runtimeMaterialInventory(true) + saved, err := json.Marshal(empty) + Expect(err).NotTo(HaveOccurred()) + Expect(string(saved)).To(ContainSubstring(`"links":[]`)) + Expect(string(saved)).To(ContainSubstring(`"search":[]`)) + Expect(string(saved)).To(ContainSubstring(`"perLink":{}`)) + Expect(string(saved)).To(ContainSubstring(`"domains":{}`)) + Expect(string(saved)).To(ContainSubstring(`"neighbors":[]`)) + var reloaded inventory.Inventory + Expect(json.Unmarshal(saved, &reloaded)).To(Succeed()) + Canonicalize(&reloaded) + Expect(reloaded.Fingerprint).To(Equal(fresh.Fingerprint)) + Expect(Diff(&fresh, &reloaded)).To(BeEmpty()) + Expect(reloaded.DPUs[0].Runtime.Network.Resolver.Domains).NotTo(BeNil()) + Expect(reloaded.Hosts[0].Runtime.Observations["resolver-dns"].ReasonCode).To(Equal("TOOL_NOT_AVAILABLE")) + }) + + It("preserves absence of whole Network and Hardware observations", func() { + without := runtimeMaterialInventory(false) + without.DPUs[0].Runtime.Network = nil + without.DPUs[0].Runtime.Hardware = nil + withEmpty := runtimeMaterialInventory(false) + withEmpty.DPUs[0].Runtime.Network = &inventory.NetworkObservation{} + withEmpty.DPUs[0].Runtime.Hardware = &inventory.HardwareObservation{} + Canonicalize(&without) + Canonicalize(&withEmpty) + Expect(withEmpty.Fingerprint).NotTo(Equal(without.Fingerprint)) + Expect(Diff(&without, &withEmpty)).NotTo(BeEmpty()) + }) + + It("preserves resolver map keys and real resolver values", func() { + without := runtimeMaterialInventory(false) + withFact := runtimeMaterialInventory(false) + withFact.DPUs[0].Runtime.Network.Resolver.PerLink = map[string][]string{"evidence": {"192.0.2.53"}} + before, err := json.Marshal(withFact.DPUs[0].Runtime.Network.Resolver.PerLink) + Expect(err).NotTo(HaveOccurred()) + Canonicalize(&without) + Canonicalize(&withFact) + Expect(withFact.Fingerprint).NotTo(Equal(without.Fingerprint)) + Expect(Diff(&without, &withFact)).NotTo(BeEmpty()) + after, err := json.Marshal(withFact.DPUs[0].Runtime.Network.Resolver.PerLink) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(before), "material normalization must retain arbitrary map keys and values") + }) +}) + +func runtimeMaterialInventory(empty bool) inventory.Inventory { + inv := estate() + inv.FormatVersion = inventory.FormatVersion + inv.DPUs = inv.DPUs[:1] + dpu := &inv.DPUs[0] + dpu.AdvertisedResetTypes = nil + functions := []inventory.PCIFunction{{Address: "0000:01:00.0", Driver: "mlx5_core"}} + adapterFunctions := []inventory.PCIFunction(nil) + if empty { + dpu.AdvertisedResetTypes = []string{} + adapterFunctions = []inventory.PCIFunction{} + functions[0].Netdevs = []string{} + } + dpu.Adapters = []inventory.Adapter{{Key: "adapter-a", SerialNumber: "CARD-A", SerialNumberRaw: "CARD-A", Model: "BlueField", IdentityState: "known", Selection: "selected", PCIFunctions: adapterFunctions}, {Key: "adapter-b", SerialNumber: "CARD-B", SerialNumberRaw: "CARD-B", Model: "BlueField", IdentityState: "known", Selection: "selected", PCIFunctions: functions}, {Key: "adapter-c", SerialNumber: "CARD-C", SerialNumberRaw: "CARD-C", Model: "BlueField", IdentityState: "known", Selection: "selected", PCIFunctions: []inventory.PCIFunction{{Address: "0000:02:00.0", Netdevs: []string{"ens6f0", "ens5f0"}}}}} + dpuRuntime := runtimeMaterialObservation(empty) + dpu.Runtime = &dpuRuntime + hostRuntime := runtimeMaterialObservation(empty) + inv.Hosts[0].Runtime = &hostRuntime + return inv +} + +func runtimeMaterialObservation(empty bool) inventory.RuntimeObservation { + interfaceAddresses := []string(nil) + interfaces := []inventory.Interface{{Name: "data1", Index: 5, MTU: 9000, Addresses: interfaceAddresses}, {Name: "data0", Index: 4, MTU: 9000, Addresses: []string{"192.0.2.44"}}} + var routes []inventory.Route + var dns []string + network := &inventory.NetworkObservation{ + Resolver: inventory.ResolverObservation{Owner: "systemd-resolved"}, + } + if empty { + interfaces[0].Addresses = []string{} + routes = []inventory.Route{} + dns = []string{} + network.Links = []inventory.NetworkLink{} + network.Resolver.Search = []string{} + network.Resolver.PerLink = map[string][]string{} + network.Resolver.Domains = map[string][]string{} + network.Resolver.DNSMasqServers = []string{} + network.Neighbors = []inventory.Neighbor{} + network.LLDP = []inventory.LLDPNeighbor{} + } + return inventory.RuntimeObservation{ + Interfaces: interfaces, + Routes: routes, + DNS: dns, + Observations: map[string]inventory.Observation{ + "interfaces": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "ip"}, + "routes": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "ip"}, + "dns": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "cat"}, + "network-links": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "ip"}, + "neighbors": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "ip"}, + "resolver-dns": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "resolvectl"}, + "resolver-domains": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "resolvectl"}, + "dnsmasq": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "sh"}, + "lldp": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "lldpctl"}, + }, + Network: network, + } +} diff --git a/pkg/dpf/discovery/software.go b/pkg/dpf/discovery/software.go new file mode 100644 index 00000000..2c445d22 --- /dev/null +++ b/pkg/dpf/discovery/software.go @@ -0,0 +1,261 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "slices" + "strconv" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" +) + +// SoftwareChecks reports release compatibility separately from discovery success. +// Only exact catalog models and identity-bound measurements can satisfy a minimum. +func SoftwareChecks(inv *inventory.Inventory) []inventory.Check { + out := []inventory.Check{} + if inv == nil || inv.Software == nil { + return out + } + for _, d := range inv.DPUs { + if !softwareIdentityCurrent(inv, d) { + out = append(out, inventory.Check{ID: "dpf.software.applicability", Subject: d.RecordKey, Vantage: "executor", RequiredPhase: "generate", State: "unknown", ReasonCode: "SOFTWARE_IDENTITY_UNAVAILABLE", Message: "Software compatibility requires a successful current observation of a unique DPU identity; inspect this device's identity observation reason.", EvidenceRefs: slices.Clone(d.EvidenceRefs)}) + continue + } + host := softwareHost(inv, d) + model := d.Model + if d.Runtime != nil { + board := d.Runtime.Observations["board-product"] + if board.State == "observed" && len(board.Values) == 1 { + model = board.Values[0] + } + } + hw, ok := inv.Software.RequirementsFor(model) + base := inventory.Check{ID: "dpf.software.applicability", Subject: d.RecordKey, Vantage: "executor", RequiredPhase: "generate", State: "unknown", ReasonCode: "BOM_MODEL_UNSUPPORTED", Message: "The selected release BOM has no exact applicability entry for this observed model; software compatibility remains unverified.", EvidenceRefs: slices.Clone(d.EvidenceRefs)} + if !ok { + out = append(out, base) + continue + } + base.State = "passed" + base.ReasonCode = "BOM_MODEL_MATCH" + base.Message = "Observed model matches a documented BOM entry; hardware qualification remains separate." + out = append(out, base) + for _, req := range hw.Requirements { + // Target coordinates describe desired software, not a minimum + // readiness predicate. Unknown constraints still explain the gap. + if req.State == "known" && req.Minimum == "" { + continue + } + c := inventory.Check{ID: "dpf.software." + req.Component, Subject: d.RecordKey, Vantage: "executor", RequiredPhase: "generate", State: "unknown", ReasonCode: req.ReasonCode, Message: req.Message, EvidenceRefs: []string{}} + if req.State != "known" { + out = append(out, c) + continue + } + values, refs, issue := softwareVersions(d, host, req) + c.EvidenceRefs = append(c.EvidenceRefs, refs...) + if issue.ReasonCode == "SOFTWARE_VERSION_CONFLICT" { + c.ReasonCode, c.Message = issue.ReasonCode, issue.Message + out = append(out, c) + continue + } + slices.Sort(values) + values = slices.Compact(values) + switch len(values) { + case 0: + c.ReasonCode = "SOFTWARE_VERSION_UNAVAILABLE" + c.Message = "No running or current, identity-bound version was measured for this requirement; inspect firmware/tool observation reasons." + if issue.ReasonCode != "" { + c.ReasonCode, c.Message = issue.ReasonCode, issue.Message + } + case 1: + cmp, valid := compareVersion(values[0], req.Minimum) + if req.Minimum == "" || !valid { + c.ReasonCode = "SOFTWARE_VERSION_UNCOMPARABLE" + c.Message = "The observed version cannot be safely compared with the declared minimum." + } else if cmp < 0 { + c.State = "failed" + c.ReasonCode = "SOFTWARE_BELOW_MINIMUM" + c.Message = "Observed version " + values[0] + " is below the selected release minimum " + req.Minimum + "." + } else { + c.State = "passed" + c.ReasonCode = "SOFTWARE_MINIMUM_MET" + c.Message = "The measured version satisfies the declared minimum; this is not overall deployment qualification." + } + default: + c.ReasonCode = "SOFTWARE_VERSION_CONFLICT" + c.Message = "Identity-bound sources report conflicting versions; no version was selected to satisfy the requirement." + } + out = append(out, c) + } + } + return out +} + +// A failed final identity check invalidates measurements retained for diagnosis. +// Duplicate identities must not become software-compatibility evidence either. +func softwareIdentityCurrent(inv *inventory.Inventory, d inventory.Device) bool { + if d.ObservationState != "observed" || d.ID == "" || d.SerialNumber == "" { + return false + } + count := 0 + for _, other := range inv.DPUs { + if other.ID == d.ID || other.SerialNumber == d.SerialNumber { + count++ + } + } + for _, host := range inv.Hosts { + if host.ID == d.ID { + count++ + } + } + return count == 1 +} + +// Host measurements are eligible only through a unique accepted association. +// A coincidentally matching serial elsewhere in the inventory is insufficient. +func softwareHost(inv *inventory.Inventory, d inventory.Device) *inventory.Device { + var association *inventory.Association + for i := range inv.Associations { + a := &inv.Associations[i] + if a.DPUID != d.ID { + continue + } + if association != nil || a.HostID == "" || a.HostAdapterKey == "" || a.SerialNumber != d.SerialNumber { + return nil + } + association = a + } + if association == nil { + return nil + } + var host *inventory.Device + for i := range inv.Hosts { + candidate := &inv.Hosts[i] + if candidate.ID != association.HostID { + continue + } + if host != nil || candidate.ObservationState != "observed" { + return nil + } + host = candidate + } + if host == nil { + return nil + } + for _, other := range inv.DPUs { + if other.ID == host.ID { + return nil + } + } + matches := 0 + for _, adapter := range host.Adapters { + if adapter.SerialNumber != d.SerialNumber { + continue + } + if adapter.Key != association.HostAdapterKey { + return nil + } + matches++ + } + if matches != 1 { + return nil + } + return host +} + +func softwareVersions(d inventory.Device, host *inventory.Device, req releases.Requirement) ([]string, []string, inventory.Observation) { + values, refs := []string{}, []string{} + issue := inventory.Observation{} + source := d.Observations["firmware-inventory"] + for _, fw := range d.Firmware { + if fw.Component != req.Component { + continue + } + if fw.State == "conflict" || fw.ActivationReasonCode == "ACTIVATION_CONFLICT" { + refs = append(refs, fw.EvidenceRefs...) + issue = inventory.Observation{State: "unknown", ReasonCode: "SOFTWARE_VERSION_CONFLICT", Message: "Component version or activation evidence conflicts; no alternate source can satisfy this requirement until the conflict is resolved."} + } + if source.State == "observed" && fw.State == "known" && fw.Activation == "running" { + values = append(values, fw.RawVersion) + refs = append(refs, fw.EvidenceRefs...) + } + } + if req.Component == "nic" { + for _, device := range []*inventory.Device{&d, host} { + if device == nil || device.Runtime == nil || device.Runtime.Hardware == nil { + continue + } + for _, card := range device.Runtime.Hardware.Cards { + if card.SerialNumber == d.SerialNumber && card.Firmware.State == "observed" { + values = append(values, card.Firmware.Values...) + refs = append(refs, card.EvidenceRefs...) + } + } + } + } + if req.Scope == "host" && req.Component == "mft" && host != nil && host.Runtime != nil { + version := host.Runtime.Observations["mft-version"] + if version.State == "observed" { + values = append(values, version.Values...) + refs = append(refs, host.EvidenceRefs...) + } + } + if len(values) == 0 && issue.ReasonCode == "" && source.State != "observed" && source.ReasonCode != "" && req.Scope != "host" { + issue = inventory.Observation{State: "unknown", ReasonCode: source.ReasonCode, Message: "The Redfish firmware inventory is unavailable or incomplete: " + source.Message} + } + return values, refs, issue +} + +// Version grammar is deliberately narrow: numeric release components separated by +// dots and an optional numeric build suffix. Vendor strings require normalization +// by their specific collector before they can satisfy a minimum. +func compareVersion(a, b string) (int, bool) { + parse := func(v string) ([]uint64, bool) { + if v == "" { + return nil, false + } + out := []uint64{} + for _, s := range strings.Split(strings.ReplaceAll(v, "-", "."), ".") { + if s == "" { + return nil, false + } + for _, r := range s { + if r < '0' || r > '9' { + return nil, false + } + } + n, e := strconv.ParseUint(s, 10, 64) + if e != nil { + return nil, false + } + out = append(out, n) + } + return out, true + } + aa, ok := parse(a) + if !ok { + return 0, false + } + bb, ok := parse(b) + if !ok { + return 0, false + } + n := max(len(aa), len(bb)) + for i := 0; i < n; i++ { + var x, y uint64 + if i < len(aa) { + x = aa[i] + } + if i < len(bb) { + y = bb[i] + } + if x < y { + return -1, true + } + if x > y { + return 1, true + } + } + return 0, true +} diff --git a/pkg/dpf/discovery/software_integration_test.go b/pkg/dpf/discovery/software_integration_test.go new file mode 100644 index 00000000..217b5a70 --- /dev/null +++ b/pkg/dpf/discovery/software_integration_test.go @@ -0,0 +1,130 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("Redfish collector to software BOM contract", func() { + It("uses actual running activation and rejects staged, conflicting, partial and replaced observations", func() { + Expect(os.Setenv("DPF_SOFTWARE_TEST_PASSWORD", "fixture-password")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_SOFTWARE_TEST_PASSWORD") + dir, err := os.MkdirTemp("", "dpf-software-") + Expect(err).NotTo(HaveOccurred()) + DeferCleanup(os.RemoveAll, dir) + for _, scenario := range []struct { + name, activation, expected string + partial, replace bool + }{ + {name: "running", activation: "ActiveTargets", expected: "SOFTWARE_MINIMUM_MET"}, + {name: "staged", activation: "StagedTargets", expected: "SOFTWARE_VERSION_UNAVAILABLE"}, + {name: "conflicting activation", activation: "both", expected: "SOFTWARE_VERSION_CONFLICT"}, + {name: "partial collection", activation: "ActiveTargets", partial: true, expected: "ACCESS_DENIED"}, + {name: "identity replaced", activation: "ActiveTargets", replace: true, expected: "SOFTWARE_IDENTITY_UNAVAILABLE"}, + } { + cardReads := 0 + writes := []string{} + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + writes = append(writes, r.Method+" "+r.URL.Path) + } + link := func(uri string) redfish.Document { return redfish.Document{"@odata.id": uri} } + var doc redfish.Document + switch r.URL.Path { + case "/redfish/v1/": + doc = redfish.Document{"SessionService": link("/redfish/v1/SessionService"), "Systems": link("/redfish/v1/Systems"), "Chassis": link("/redfish/v1/Chassis"), "UpdateService": link("/redfish/v1/UpdateService")} + case "/redfish/v1/SessionService": + doc = redfish.Document{"Sessions": link("/redfish/v1/SessionService/Sessions")} + case "/redfish/v1/SessionService/Sessions": + w.Header().Set("X-Auth-Token", "fixture-token") + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/owned") + w.WriteHeader(http.StatusCreated) + doc = redfish.Document{} + case "/redfish/v1/SessionService/Sessions/owned": + w.WriteHeader(http.StatusNoContent) + return + case "/redfish/v1/Systems": + doc = redfish.Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Systems/DPU")}} + case "/redfish/v1/Systems/DPU": + doc = redfish.Document{"UUID": "11111111-1111-4111-8111-111111111111", "Manufacturer": "NVIDIA", "Model": "BlueField-3"} + case "/redfish/v1/Chassis": + doc = redfish.Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Chassis/Card")}} + case "/redfish/v1/Chassis/Card": + cardReads++ + serial := "CARD-A" + if scenario.replace && cardReads > 1 { + serial = "CARD-B" + } + doc = redfish.Document{"Model": "BlueField-3", "SerialNumber": serial, "Links": redfish.Document{"ComputerSystems": []any{link("/redfish/v1/Systems/DPU")}}} + case "/redfish/v1/UpdateService": + doc = redfish.Document{"FirmwareInventory": link("/redfish/v1/UpdateService/FirmwareInventory")} + case "/redfish/v1/UpdateService/FirmwareInventory": + members := []any{link("/redfish/v1/UpdateService/FirmwareInventory/NIC")} + if scenario.partial { + members = append(members, link("/redfish/v1/UpdateService/FirmwareInventory/Denied")) + } + doc = redfish.Document{"Members@odata.count": len(members), "Members": members} + case "/redfish/v1/UpdateService/FirmwareInventory/NIC": + links := redfish.Document{scenario.activation: []any{link("/redfish/v1/Systems/DPU")}} + if scenario.activation == "both" { + links = redfish.Document{"ActiveTargets": []any{link("/redfish/v1/Systems/DPU")}, "StagedTargets": []any{link("/redfish/v1/Systems/DPU")}} + } + doc = redfish.Document{"Id": "NIC_Firmware", "Name": "NIC Firmware", "Version": "32.38.1002", "Links": links} + case "/redfish/v1/UpdateService/FirmwareInventory/Denied": + w.WriteHeader(http.StatusForbidden) + return + default: + w.WriteHeader(http.StatusNotFound) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(doc) + })) + source := fmt.Sprintf("management:\n kubeconfigRef: {file: unused}\ncredentials:\n read: {username: reader, passwordRef: {env: DPF_SOFTWARE_TEST_PASSWORD}}\ndiscovery:\n tls: {insecureSkipVerify: true}\n dpus: [{bmcEndpoint: '%s', credentialRef: read}]\n hosts: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: read}]\n", server.URL) + in, e := config.Decode([]byte(source), filepath.Join(dir, "site.yaml")) + Expect(e).NotTo(HaveOccurred()) + request, e := config.Resolve(in) + Expect(e).NotTo(HaveOccurred()) + d, _, e := (redfish.Collector{}).Collect(context.Background(), request.Targets[0], time.Second) + if scenario.replace { + Expect(redfish.Code(e)).To(Equal("IDENTITY_CHANGED")) + } else { + Expect(e).NotTo(HaveOccurred()) + } + server.Close() + Expect(writes).To(Equal([]string{"POST /redfish/v1/SessionService/Sessions", "DELETE /redfish/v1/SessionService/Sessions/owned"})) + // Exact board product is independent SSH evidence; a generic Redfish family + // is intentionally insufficient for production BOM applicability. + d.Runtime = &inventory.RuntimeObservation{Observations: map[string]inventory.Observation{"board-product": {State: "observed", Values: []string{"B3210"}}}} + release, e := releases.Resolve("") + Expect(e).NotTo(HaveOccurred()) + inv := &inventory.Inventory{Software: &release, DPUs: []inventory.Device{d}} + component := "nic" + if scenario.replace { + component = "applicability" + } + check := softwareCheck(SoftwareChecks(inv), component) + Expect(check.ReasonCode).To(Equal(scenario.expected), scenario.name) + if scenario.name == "running" { + Expect(check.State).To(Equal("passed")) + } else { + Expect(check.State).To(Equal("unknown")) + } + } + }) +}) diff --git a/pkg/dpf/discovery/software_test.go b/pkg/dpf/discovery/software_test.go new file mode 100644 index 00000000..1f4b1026 --- /dev/null +++ b/pkg/dpf/discovery/software_test.go @@ -0,0 +1,149 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package discovery + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("Software BOM readiness", func() { + It("does not confuse inventory collection, software applicability and hardware qualification", func() { + r, err := releases.Resolve("") + Expect(err).NotTo(HaveOccurred()) + inv := &inventory.Inventory{Software: &r, DPUs: []inventory.Device{{RecordKey: "bf4", ID: "bf4", SerialNumber: "CARD4", ObservationState: "observed", Model: "BlueField-4"}}} + checks := SoftwareChecks(inv) + Expect(checks).To(HaveLen(1)) + Expect(checks[0].ReasonCode).To(Equal("BOM_MODEL_UNSUPPORTED")) + Expect(checks[0].RequiredPhase).To(Equal("generate")) + }) + It("requires active evidence and retains unknown requirements", func() { + r, err := releases.Resolve("") + Expect(err).NotTo(HaveOccurred()) + d := inventory.Device{RecordKey: "bf3", ID: "bf3", ObservationState: "observed", Observations: map[string]inventory.Observation{"firmware-inventory": {State: "observed"}}, Model: "B3210", SerialNumber: "MT1234567890", Firmware: []inventory.Firmware{{Component: "nic", State: "known", RawVersion: "32.38.1002", Activation: "unknown"}}} + inv := &inventory.Inventory{Software: &r, DPUs: []inventory.Device{d}} + find := func() inventory.Check { + for _, c := range SoftwareChecks(inv) { + if c.ID == "dpf.software.nic" { + return c + } + } + Fail("missing NIC check") + return inventory.Check{} + } + Expect(find().ReasonCode).To(Equal("SOFTWARE_VERSION_UNAVAILABLE")) + inv.DPUs[0].Firmware[0].Activation = "running" + Expect(find().State).To(Equal("passed")) + inv.DPUs[0].Firmware[0].RawVersion = "32.37.9999" + Expect(find().State).To(Equal("failed")) + inv.DPUs[0].Firmware = append(inv.DPUs[0].Firmware, inventory.Firmware{Component: "nic", State: "known", RawVersion: "32.38.1002", Activation: "running"}) + Expect(find().ReasonCode).To(Equal("SOFTWARE_VERSION_CONFLICT")) + for _, c := range SoftwareChecks(inv) { + if c.State != "passed" { + Expect(c.ReasonCode).NotTo(BeEmpty()) + Expect(c.Message).NotTo(BeEmpty()) + } + } + }) + It("accepts only current identity-bound source measurements", func() { + for _, scenario := range []struct { + name string + mutate func(*inventory.Inventory) + want string + }{ + {"accepted host", func(*inventory.Inventory) {}, "passed"}, + {"failed DPU", func(i *inventory.Inventory) { + i.DPUs[0].ObservationState = "unavailable" + i.DPUs[0].ReasonCode = "IDENTITY_CHANGED" + }, "identity"}, + {"missing associated adapter", func(i *inventory.Inventory) { i.Hosts[0].Adapters = nil }, "unknown"}, + {"failed host", func(i *inventory.Inventory) { i.Hosts[0].ObservationState = "unavailable" }, "unknown"}, + {"unassociated host", func(i *inventory.Inventory) { i.Associations = nil }, "unknown"}, + {"duplicate association", func(i *inventory.Inventory) { i.Associations = append(i.Associations, i.Associations[0]) }, "unknown"}, + {"duplicate DPU", func(i *inventory.Inventory) { i.DPUs = append(i.DPUs, i.DPUs[0]) }, "identity"}, + {"unrelated serial", func(i *inventory.Inventory) { i.Hosts[0].Runtime.Hardware.Cards[0].SerialNumber = "OTHER" }, "unknown"}, + } { + inv := softwareHostFixture() + scenario.mutate(inv) + checks := SoftwareChecks(inv) + if scenario.want == "identity" { + for _, c := range checks { + Expect(c.ReasonCode).To(Equal("SOFTWARE_IDENTITY_UNAVAILABLE"), scenario.name) + Expect(c.State).To(Equal("unknown")) + } + } else { + Expect(softwareCheck(checks, "nic").State).To(Equal(scenario.want), scenario.name) + } + } + }) + It("keeps explicit firmware and activation conflicts dominant over valid SSH", func() { + for _, fw := range []inventory.Firmware{ + {Component: "nic", State: "conflict", ReasonCode: "FIRMWARE_VERSION_CONFLICT"}, + {Component: "nic", State: "known", RawVersion: "32.38.1002", Activation: "unknown", ActivationReasonCode: "ACTIVATION_CONFLICT"}, + } { + inv := softwareHostFixture() + inv.DPUs[0].Firmware = []inventory.Firmware{fw} + check := softwareCheck(SoftwareChecks(inv), "nic") + Expect(check.State).To(Equal("unknown")) + Expect(check.ReasonCode).To(Equal("SOFTWARE_VERSION_CONFLICT")) + } + }) + It("retains incomplete source reasons and accepts an independent associated SSH source", func() { + inv := softwareHostFixture() + d := &inv.DPUs[0] + d.Observations = map[string]inventory.Observation{"firmware-inventory": {State: "unavailable", ReasonCode: "ACCESS_DENIED", Message: "A firmware member returned HTTP 403."}} + d.Firmware = []inventory.Firmware{{Component: "nic", State: "known", RawVersion: "32.38.1002", Activation: "running", EvidenceRefs: []string{"partial-redfish"}}} + check := softwareCheck(SoftwareChecks(inv), "nic") + Expect(check.State).To(Equal("passed")) + Expect(check.EvidenceRefs).To(ContainElement("associated-ssh")) + Expect(check.EvidenceRefs).NotTo(ContainElement("partial-redfish")) + Expect(d.Observations["firmware-inventory"].ReasonCode).To(Equal("ACCESS_DENIED")) + inv.Associations = nil + check = softwareCheck(SoftwareChecks(inv), "nic") + Expect(check.State).To(Equal("unknown")) + Expect(check.ReasonCode).To(Equal("ACCESS_DENIED")) + Expect(check.Message).To(ContainSubstring("403")) + }) + It("does not turn a desired target into a compatibility predicate", func() { + inv := softwareHostFixture() + inv.Software.Hardware[0].Requirements = append(inv.Software.Hardware[0].Requirements, releases.Requirement{Component: "desired-only", State: "known", Target: "1.2.3"}) + for _, c := range SoftwareChecks(inv) { + Expect(c.ID).NotTo(Equal("dpf.software.desired-only")) + } + }) + It("compares numeric build versions without ordering vendor text", func() { + c, ok := compareVersion("4.33.0-169", "4.33.0-168") + Expect(ok).To(BeTrue()) + Expect(c).To(Equal(1)) + _, ok = compareVersion("unverified", "4.33.0-169") + Expect(ok).To(BeFalse()) + }) + It("explains every nonpassing acceptance predicate", func() { + for _, code := range []string{"EXACTLY_THREE_CONTROL_PLANES", "PHYSICAL_IDENTITY_REQUIRED", "QUALIFIED_ADAPTER_REQUIRED", "DPU_IDENTITY_REQUIRED", "UNIQUE_PHYSICAL_MATCH_REQUIRED", "REDFISH_AUTHORIZATION_NOT_EVALUATED", "DPF_AUTHENTICATION_NOT_EVALUATED", "INVENTORY_MODE", "EXACT_NETWORK_COVERAGE_REQUIRED"} { + c := inventory.Check{State: "unknown", ReasonCode: code} + ExplainCheck(&c) + Expect(c.Message).NotTo(BeEmpty()) + } + }) +}) + +func softwareCheck(checks []inventory.Check, component string) inventory.Check { + for _, c := range checks { + if c.ID == "dpf.software."+component { + return c + } + } + Fail("missing software check for " + component) + return inventory.Check{} +} +func softwareHostFixture() *inventory.Inventory { + r, err := releases.Resolve("") + Expect(err).NotTo(HaveOccurred()) + return &inventory.Inventory{Software: &r, + DPUs: []inventory.Device{{RecordKey: "card", ID: "dpu-a", SerialNumber: "CARD-A", ObservationState: "observed", Model: "B3210"}}, + Hosts: []inventory.Device{{ID: "host-a", ObservationState: "observed", Adapters: []inventory.Adapter{{Key: "adapter-a", SerialNumber: "CARD-A"}}, Runtime: &inventory.RuntimeObservation{Hardware: &inventory.HardwareObservation{Cards: []inventory.CardHardware{{SerialNumber: "CARD-A", Firmware: inventory.Observation{State: "observed", Values: []string{"32.38.1002"}}, EvidenceRefs: []string{"associated-ssh"}}}}}}}, + Associations: []inventory.Association{{DPUID: "dpu-a", HostID: "host-a", SerialNumber: "CARD-A", HostAdapterKey: "adapter-a"}}, + } +} diff --git a/pkg/dpf/discovery/vendor_integration_test.go b/pkg/dpf/discovery/vendor_integration_test.go new file mode 100644 index 00000000..20f09cbc --- /dev/null +++ b/pkg/dpf/discovery/vendor_integration_test.go @@ -0,0 +1,179 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "context" + "encoding/json" + "encoding/pem" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sync/atomic" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/artifacts" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// Only the DPU side is fixed: the host uses the real HTTPS/session/ownership +// collector, so acceptance depends on normalized HPE PCI evidence. +type vendorBindingCollector struct{} + +func (vendorBindingCollector) Collect(ctx context.Context, target config.Target, timeout time.Duration) (inventory.Device, map[string]inventory.Evidence, error) { + if target.Role == "hosts" { + return (redfish.Collector{}).Collect(ctx, target, timeout) + } + return inventory.Device{ + ID: "dpu-one", RecordKey: target.InputRef, InputRef: target.InputRef, + ObservationState: "observed", Qualification: "unqualified", InventoryCompleteness: "partial", + SerialNumber: "CARD-ONE", BMCEndpoint: target.Endpoint, + Observations: map[string]inventory.Observation{"bmc-administrator": {State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED", Message: "fixture verified"}}, + }, nil, nil +} + +var _ = Describe("HPE PCI vendor evidence through discovery publication", func() { + It("publishes NVIDIA and unknown vendors but preserves the handoff after a same-serial vendor contradiction", func() { + var vendor atomic.Int64 + vendor.Store(0x15b3) + link := func(uri string) redfish.Document { return redfish.Document{"@odata.id": uri} } + collection := func(uris ...string) redfish.Document { + members := []any{} + for _, uri := range uris { + members = append(members, link(uri)) + } + return redfish.Document{"Members@odata.count": len(uris), "Members": members} + } + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + const sessions = "/redfish/v1/SessionService/Sessions" + if r.Method == http.MethodPost && r.URL.Path == sessions { + w.Header().Set("X-Auth-Token", "fixture-session") + w.Header().Set("Location", sessions+"/owned") + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{}`)) + return + } + if r.Method == http.MethodDelete && r.URL.Path == sessions+"/owned" { + w.WriteHeader(http.StatusNoContent) + return + } + if r.Method != http.MethodGet { + w.WriteHeader(http.StatusMethodNotAllowed) + return + } + var doc redfish.Document + switch r.URL.Path { + case "/redfish/v1/": + doc = redfish.Document{"SessionService": link("/redfish/v1/SessionService"), "Systems": link("/redfish/v1/Systems")} + case "/redfish/v1/SessionService": + doc = redfish.Document{"Sessions": link(sessions)} + case "/redfish/v1/Systems": + doc = collection("/redfish/v1/Systems/host") + case "/redfish/v1/Systems/host": + doc = redfish.Document{"UUID": "11111111-1111-4111-8111-111111111111", "Manufacturer": "HPE", "Model": "ProLiant", "Links": redfish.Document{"Chassis": []any{link("/redfish/v1/Chassis/host")}}, "Oem": redfish.Document{"Hpe": redfish.Document{"Links": redfish.Document{"PCIDevices": []any{link("/redfish/v1/Systems/host/PCIDevices")}}}}} + case "/redfish/v1/Chassis/host": + doc = redfish.Document{"NetworkAdapters": link("/redfish/v1/Chassis/host/NetworkAdapters")} + case "/redfish/v1/Chassis/host/NetworkAdapters": + doc = collection("/redfish/v1/Chassis/host/NetworkAdapters/selected", "/redfish/v1/Chassis/host/NetworkAdapters/other") + case "/redfish/v1/Chassis/host/NetworkAdapters/selected": + doc = redfish.Document{"Id": "selected", "SerialNumber": "CARD-ONE", "Model": "Network adapter"} + case "/redfish/v1/Chassis/host/NetworkAdapters/other": + doc = redfish.Document{"Id": "other", "SerialNumber": "OTHER-NIC", "Model": "Intel E810"} + case "/redfish/v1/Systems/host/PCIDevices": + doc = collection("/redfish/v1/Systems/host/PCIDevices/selected", "/redfish/v1/Systems/host/PCIDevices/other") + case "/redfish/v1/Systems/host/PCIDevices/selected": + doc = redfish.Document{"ClassCode": 2, "DeviceResourceId": "selected", "SegmentNumber": 0, "BusNumber": 129, "DeviceNumber": 0, "FunctionNumber": 0, "DeviceID": 41695} + if value := vendor.Load(); value >= 0 { + doc["VendorID"] = value + } + case "/redfish/v1/Systems/host/PCIDevices/other": + doc = redfish.Document{"ClassCode": 2, "DeviceResourceId": "other", "VendorID": 0x8086} + default: + w.WriteHeader(http.StatusNotFound) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(doc) + })) + DeferCleanup(server.Close) + dir, err := filepath.EvalSymlinks(GinkgoT().TempDir()) + Expect(err).NotTo(HaveOccurred()) + caPath := filepath.Join(dir, "ca.pem") + Expect(os.WriteFile(caPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw}), 0600)).To(Succeed()) + Expect(os.Setenv("DPF_VENDOR_BINDING_PASSWORD", "fixture-password")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_VENDOR_BINDING_PASSWORD") + seed := fmt.Sprintf("credentials:\n reader: {username: reader, passwordRef: {env: DPF_VENDOR_BINDING_PASSWORD}}\ndiscovery:\n tls: {caFile: '%s'}\n hosts: [{bmcEndpoint: '%s', credentialRef: reader}]\n dpus: [{bmcEndpoint: 'https://dpu.invalid', credentialRef: reader}]\n probes: {mode: inventory}\n", caPath, server.URL) + inputPath := filepath.Join(dir, "seed.yaml") + Expect(os.WriteFile(inputPath, []byte(seed), 0600)).To(Succeed()) + in, err := config.Load(inputPath) + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(in) + Expect(err).NotTo(HaveOccurred()) + output := filepath.Join(dir, "discovered.yaml") + var previous []byte + for _, value := range []int64{0x15b3, -1, 0x8086} { + vendor.Store(value) + store, err := artifacts.New(in, output, true) + Expect(err).NotTo(HaveOccurred()) + report := Collect(context.Background(), req, store.RunID(), Dependencies{BMC: vendorBindingCollector{}}) + Expect(report.Errors).To(BeEmpty()) + Expect(report.Inventory.Hosts[0].Adapters).To(HaveLen(2)) + var selected inventory.Adapter + for _, adapter := range report.Inventory.Hosts[0].Adapters { + if adapter.SerialNumber == "CARD-ONE" { + selected = adapter + } else { + Expect(adapter.VendorID).To(Equal("8086")) + Expect(adapter.Selection).To(Equal("not-selected")) + } + } + Expect(selected.PCIAddress).To(Equal("0000:81:00.0")) + Expect(store.Publish(context.Background(), &report)).To(Succeed()) + current, err := os.ReadFile(output) + Expect(err).NotTo(HaveOccurred()) + if value == 0x8086 { + Expect(selected.VendorID).To(Equal("8086")) + Expect(selected.Selection).To(Equal("not-selected")) + Expect(ExitCode(report)).To(Equal(2)) + Expect(report.Discovery.Outcome).To(Equal("partial")) + Expect(report.Inventory.Associations).To(BeEmpty()) + Expect(current).To(Equal(previous)) + var check inventory.Check + for _, c := range report.Discovery.Checks { + if c.ReasonCode == "HOST_ADAPTER_VENDOR_CONFLICT" { + check = c + } + } + Expect(check.State).To(Equal("failed")) + Expect(check.Message).To(ContainSubstring("Verify the selected endpoints")) + Expect(check.EvidenceRefs).To(BeEmpty()) + Expect(selected.EvidenceRefs).To(BeEmpty()) + Expect(report.Inventory.Evidence).To(BeEmpty()) + Expect(report.Artifacts).To(ContainElement(HaveField("Path", filepath.Join(store.RunDir, "partial.yaml")))) + persisted, err := os.ReadFile(filepath.Join(store.RunDir, "report.json")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(persisted)).To(ContainSubstring("HOST_ADAPTER_VENDOR_CONFLICT")) + Expect(string(persisted)).NotTo(ContainSubstring(`"evidenceRefs"`)) + Expect(string(persisted)).NotTo(ContainSubstring(`"evidence"`)) + Expect(string(persisted)).NotTo(ContainSubstring("fixture-password")) + } else { + if value < 0 { + Expect(selected.VendorID).To(BeEmpty()) + } else { + Expect(selected.VendorID).To(Equal("15b3")) + } + Expect(ExitCode(report)).To(Equal(0)) + Expect(report.Discovery.Outcome).To(Equal("complete")) + Expect(report.Inventory.Associations).To(HaveLen(1)) + previous = current + } + } + }) +}) diff --git a/pkg/dpf/inventory/addresses.go b/pkg/dpf/inventory/addresses.go new file mode 100644 index 00000000..503147cd --- /dev/null +++ b/pkg/dpf/inventory/addresses.go @@ -0,0 +1,17 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package inventory + +// AddressCandidate is an unverified locator, never physical identity or a connection grant. +type AddressCandidate struct { + Name string `json:"name" yaml:"name"` + Role string `json:"role" yaml:"role"` + MAC string `json:"mac" yaml:"mac"` + Address string `json:"address" yaml:"address"` + Source string `json:"source" yaml:"source"` + EvidenceRef string `json:"evidenceRef" yaml:"evidenceRef"` +} +type AddressLookup struct { + Candidates []AddressCandidate `json:"candidates" yaml:"candidates"` + Observations map[string]Observation `json:"observations" yaml:"observations"` +} diff --git a/pkg/dpf/inventory/config_evidence_test.go b/pkg/dpf/inventory/config_evidence_test.go new file mode 100644 index 00000000..37da7c7c --- /dev/null +++ b/pkg/dpf/inventory/config_evidence_test.go @@ -0,0 +1,97 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "encoding/json" + + "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +var _ = ginkgo.Describe("discovered configuration diagnostics", func() { + ginkgo.It("omits typed provenance fields while preserving arbitrary data map keys", func() { + device := outputDeviceFixture() + device.Runtime = &RuntimeObservation{ + Network: &NetworkObservation{Resolver: ResolverObservation{ + PerLink: map[string][]string{"evidence": {"192.0.2.10"}}, + Domains: map[string][]string{"evidenceRefs": {"example.test"}}, + }}, + } + in := Inventory{ + FormatVersion: FormatVersion, ScopeDigest: "scope", Fingerprint: "facts", + Management: &Management{APIEndpoint: "https://api.example", SystemNamespaceUID: "cluster", EvidenceRefs: []string{"management"}}, + DPUs: []Device{device}, + AddressLookup: &AddressLookup{Candidates: []AddressCandidate{{Address: "192.0.2.2", EvidenceRef: "lookup"}}}, + Probes: []ProbeResult{{NodeUID: "node", Destination: "api", NetworkMode: "pod", DNS: "passed", EvidenceRef: "probe"}}, + Evidence: map[string]Evidence{"management": {Kind: "kubernetes"}, "host": {Kind: "ssh"}, "probe": {Kind: "probe"}}, + } + before, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + data, err := yaml.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + var encoded map[string]any + Expect(yaml.Unmarshal(data, &encoded)).To(Succeed()) + Expect(encoded).NotTo(HaveKey("evidence")) + Expect(mapAt(encoded, "management")).NotTo(HaveKey("evidenceRefs")) + addressLookup := mapAt(encoded, "addressLookup") + Expect(addressLookup["candidates"].([]any)[0].(map[string]any)).NotTo(HaveKey("evidenceRef")) + Expect(encoded["probes"].([]any)[0].(map[string]any)).NotTo(HaveKey("evidenceRef")) + + resolver := encoded["dpus"].([]any)[0].(map[string]any) + resolver = mapAt(resolver, "runtime", "network", "resolver") + Expect(mapAt(resolver, "perLink")).To(HaveKeyWithValue("evidence", []any{"192.0.2.10"})) + Expect(mapAt(resolver, "domains")).To(HaveKeyWithValue("evidenceRefs", []any{"example.test"})) + var restored Inventory + Expect(yaml.Unmarshal(data, &restored)).To(Succeed()) + Expect(restored.ScopeDigest).To(Equal(in.ScopeDigest)) + Expect(restored.Fingerprint).To(Equal(in.Fingerprint)) + Expect(restored.Management.APIEndpoint).To(Equal(in.Management.APIEndpoint)) + Expect(restored.DPUs[0].SerialNumber).To(Equal(in.DPUs[0].SerialNumber)) + Expect(restored.DPUs[0].Observations["bmc-administrator"].ReasonCode).To(Equal("BMC_ADMIN_VERIFIED")) + Expect(restored.Probes[0].DNS).To(Equal("passed")) + Expect(restored.AddressLookup.Candidates[0].Address).To(Equal("192.0.2.2")) + Expect(restored.Evidence).To(BeEmpty()) + Expect(restored.DPUs[0].Runtime.Network.Resolver.PerLink).To(HaveKeyWithValue("evidence", []string{"192.0.2.10"})) + Expect(restored.DPUs[0].Runtime.Network.Resolver.Domains).To(HaveKeyWithValue("evidenceRefs", []string{"example.test"})) + after, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(before)) + }) + ginkgo.It("reads older evidence-bearing configs but drops bookkeeping on save", func() { + data := []byte("formatVersion: v1alpha2\nmanagement:\n apiEndpoint: https://api.example\n evidenceRefs: [old]\nevidence:\n old:\n kind: kubernetes\n") + var in Inventory + Expect(yaml.Unmarshal(data, &in)).To(Succeed()) + Expect(in.Evidence).To(HaveKey("old")) + in.Management.Observations = map[string]Observation{"evidence": {State: "observed", Values: []string{"kept"}}} + in.DPUs = []Device{{ + RecordKey: "legacy-device", EvidenceRefs: []string{"device-evidence"}, + Firmware: []Firmware{{Component: "os", EvidenceRefs: []string{"firmware-evidence"}}}, + }} + saved, err := yaml.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + var encoded map[string]any + Expect(yaml.Unmarshal(saved, &encoded)).To(Succeed()) + Expect(encoded).NotTo(HaveKey("evidence")) + management := mapAt(encoded, "management") + Expect(management).NotTo(HaveKey("evidenceRefs")) + Expect(mapAt(management, "observations")).To(HaveKey("evidence")) + legacyDPU := encoded["dpus"].([]any)[0].(map[string]any) + Expect(legacyDPU).NotTo(HaveKey("evidenceRefs")) + Expect(legacyDPU["firmware"].([]any)[0].(map[string]any)).NotTo(HaveKey("evidenceRefs")) + Expect(string(saved)).NotTo(ContainSubstring("device-evidence")) + Expect(string(saved)).NotTo(ContainSubstring("firmware-evidence")) + Expect(string(saved)).To(ContainSubstring("https://api.example")) + }) +}) + +func mapAt(root map[string]any, path ...string) map[string]any { + for _, key := range path { + value, ok := root[key].(map[string]any) + Expect(ok).To(BeTrue(), "expected mapping at %q", key) + root = value + } + return root +} diff --git a/pkg/dpf/inventory/device_output.go b/pkg/dpf/inventory/device_output.go new file mode 100644 index 00000000..4c6db278 --- /dev/null +++ b/pkg/dpf/inventory/device_output.go @@ -0,0 +1,120 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +// DeviceOutput is the v3 public host/DPU contract. ID is published only after +// physical identity rechecks succeed; an endpoint or retained serial is context, +// not a successful identity assertion. Collector bookkeeping stays private. +type DeviceOutput struct { + RecordKey string `json:"recordKey" yaml:"recordKey"` + InputRef string `json:"inputRef" yaml:"inputRef"` + Method string `json:"method,omitempty" yaml:"method,omitempty"` + ID string `json:"id,omitempty" yaml:"id,omitempty"` + BMCEndpoint string `json:"bmcEndpoint,omitempty" yaml:"bmcEndpoint,omitempty"` + SSHEndpoint string `json:"sshEndpoint,omitempty" yaml:"sshEndpoint,omitempty"` + SystemURI string `json:"systemURI,omitempty" yaml:"systemURI,omitempty"` + SerialNumber string `json:"serialNumber,omitempty" yaml:"serialNumber,omitempty"` + SystemUUID string `json:"systemUUID,omitempty" yaml:"systemUUID,omitempty"` + Manufacturer string `json:"manufacturer,omitempty" yaml:"manufacturer,omitempty"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + PSID string `json:"psid,omitempty" yaml:"psid,omitempty"` + BMCAdministratorVerified bool `json:"bmcAdministratorVerified,omitempty" yaml:"bmcAdministratorVerified,omitempty"` + FirmwareInventoryComplete bool `json:"firmwareInventoryComplete,omitempty" yaml:"firmwareInventoryComplete,omitempty"` + Firmware *FirmwareOutput `json:"firmware,omitempty" yaml:"firmware,omitempty"` + Runtime *RuntimeOutput `json:"runtime,omitempty" yaml:"runtime,omitempty"` + Adapters []AdapterOutput `json:"adapters,omitempty" yaml:"adapters,omitempty"` + AdvertisedResetTypes []string `json:"advertisedResetTypes,omitempty" yaml:"advertisedResetTypes,omitempty"` + Issues []DeviceIssue `json:"issues,omitempty" yaml:"issues,omitempty"` +} + +// DeviceIssue describes an unsuccessful read, missing fact or mismatch. It never +// stores successful parameter values or an observation success state. +type DeviceIssue struct { + Field string `json:"field" yaml:"field"` + Code string `json:"code" yaml:"code"` + Message string `json:"message,omitempty" yaml:"message,omitempty"` + Tool string `json:"tool,omitempty" yaml:"tool,omitempty"` +} + +// FirmwareOutput names only meaningful DPU components. Versions with unknown +// activation remain distinguishable from running and explicitly staged images. +// Unclassified vendor entries and host BMC firmware inventories are not exported. +type FirmwareOutput struct { + BMC []FirmwareVersionOutput `json:"bmc,omitempty" yaml:"bmc,omitempty"` + NIC []FirmwareVersionOutput `json:"nic,omitempty" yaml:"nic,omitempty"` + CEC []FirmwareVersionOutput `json:"cec,omitempty" yaml:"cec,omitempty"` + ATF []FirmwareVersionOutput `json:"atf,omitempty" yaml:"atf,omitempty"` + UEFI []FirmwareVersionOutput `json:"uefi,omitempty" yaml:"uefi,omitempty"` + ATFUEFI []FirmwareVersionOutput `json:"atfUEFI,omitempty" yaml:"atfUEFI,omitempty"` + BSP []FirmwareVersionOutput `json:"bsp,omitempty" yaml:"bsp,omitempty"` + OS []FirmwareVersionOutput `json:"os,omitempty" yaml:"os,omitempty"` + Golden []FirmwareVersionOutput `json:"golden,omitempty" yaml:"golden,omitempty"` + BMCERoT []FirmwareVersionOutput `json:"bmcERoT,omitempty" yaml:"bmcERoT,omitempty"` + CPUERoT []FirmwareVersionOutput `json:"cpuERoT,omitempty" yaml:"cpuERoT,omitempty"` +} + +type FirmwareVersionOutput struct { + Version string `json:"version" yaml:"version"` + Activation string `json:"activation,omitempty" yaml:"activation,omitempty"` +} + +type RuntimeOutput struct { + Hostname string `json:"hostname,omitempty" yaml:"hostname,omitempty"` + OS string `json:"os,omitempty" yaml:"os,omitempty"` + BoardProduct string `json:"boardProduct,omitempty" yaml:"boardProduct,omitempty"` + MFTVersion string `json:"mftVersion,omitempty" yaml:"mftVersion,omitempty"` + Interfaces []Interface `json:"interfaces,omitempty" yaml:"interfaces,omitempty"` + Routes []Route `json:"routes,omitempty" yaml:"routes,omitempty"` + DNS []string `json:"dns,omitempty" yaml:"dns,omitempty"` + Network *NetworkObservation `json:"network,omitempty" yaml:"network,omitempty"` + Hardware *HardwareOutput `json:"hardware,omitempty" yaml:"hardware,omitempty"` +} + +type AdapterOutput struct { + Key string `json:"adapterKey" yaml:"adapterKey"` + SerialNumber string `json:"serialNumber,omitempty" yaml:"serialNumber,omitempty"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + Slot string `json:"slot,omitempty" yaml:"slot,omitempty"` + PCIAddress string `json:"pciAddress,omitempty" yaml:"pciAddress,omitempty"` + VendorID string `json:"vendorID,omitempty" yaml:"vendorID,omitempty"` + DeviceID string `json:"deviceID,omitempty" yaml:"deviceID,omitempty"` + PCIFunctions []PCIFunction `json:"pciFunctions,omitempty" yaml:"pciFunctions,omitempty"` +} + +type HardwareOutput struct { + Cards []CardOutput `json:"cards,omitempty" yaml:"cards,omitempty"` + RShim *RShimServiceOutput `json:"rshim,omitempty" yaml:"rshim,omitempty"` +} + +type CardOutput struct { + PCIAddress string `json:"pciAddress" yaml:"pciAddress"` + Domain string `json:"domain" yaml:"domain"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + SerialNumber string `json:"serialNumber,omitempty" yaml:"serialNumber,omitempty"` + FirmwareVersion string `json:"firmwareVersion,omitempty" yaml:"firmwareVersion,omitempty"` + PSID string `json:"psid,omitempty" yaml:"psid,omitempty"` + NextBoot *NVConfigOutput `json:"nextBoot,omitempty" yaml:"nextBoot,omitempty"` + HostPrivileges []HostPrivilegeOutput `json:"hostPrivileges,omitempty" yaml:"hostPrivileges,omitempty"` +} + +// NextBoot describes the mlxconfig next-boot columns, never the running mode. +type NVConfigOutput struct { + InternalCPUOffloadEngine string `json:"internalCPUOffloadEngine,omitempty" yaml:"internalCPUOffloadEngine,omitempty"` + InternalCPURShim string `json:"internalCPURShim,omitempty" yaml:"internalCPURShim,omitempty"` + NumVFs *uint32 `json:"numVFs,omitempty" yaml:"numVFs,omitempty"` +} + +type HostPrivilegeOutput struct { + HostIndex uint32 `json:"hostIndex" yaml:"hostIndex"` + Level string `json:"level" yaml:"level"` + DisableRShim *bool `json:"disableRShim,omitempty" yaml:"disableRShim,omitempty"` + DisableTracer *bool `json:"disableTracer,omitempty" yaml:"disableTracer,omitempty"` + DisablePortOwner *bool `json:"disablePortOwner,omitempty" yaml:"disablePortOwner,omitempty"` + DisableCounterRead *bool `json:"disableCounterRead,omitempty" yaml:"disableCounterRead,omitempty"` +} + +type RShimServiceOutput struct { + LoadState string `json:"loadState" yaml:"loadState"` + ActiveState string `json:"activeState" yaml:"activeState"` +} diff --git a/pkg/dpf/inventory/device_output_adapter.go b/pkg/dpf/inventory/device_output_adapter.go new file mode 100644 index 00000000..560d46ed --- /dev/null +++ b/pkg/dpf/inventory/device_output_adapter.go @@ -0,0 +1,369 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "fmt" + "slices" + "sort" + "strings" +) + +// DeviceToOutput is the explicit boundary between collector observations and +// public facts. It does not mutate the collector's retained result. +func DeviceToOutput(d Device, role string) (DeviceOutput, error) { + role, err := deviceRole(role) + if err != nil { + return DeviceOutput{}, err + } + w := DeviceOutput{RecordKey: d.RecordKey, InputRef: d.InputRef, BMCEndpoint: d.BMCEndpoint, SSHEndpoint: d.SSHEndpoint, + SystemURI: d.SystemURI, SerialNumber: d.SerialNumber, SystemUUID: d.SystemUUID, Manufacturer: d.Manufacturer, + Model: d.Model, AdvertisedResetTypes: slices.Clone(d.AdvertisedResetTypes)} + switch { + case d.SSHEndpoint != "": + w.Method = "ssh" + case d.BMCEndpoint != "": + w.Method = "redfish" + case d.ReaderAdapter == "host-card-v1": + w.Method = "host-inventory" + } + if d.ObservationState == "observed" && d.ReasonCode == "" { + w.ID = d.ID + } + if w.ID == "" { + code := d.ReasonCode + if code == "" { + code = "PHYSICAL_IDENTITY_UNAVAILABLE" + } + w.Issues = append(w.Issues, DeviceIssue{Field: "identity", Code: code, Message: d.Message}) + } + if w.ID != "" && role == "dpus" && w.Method == "redfish" && w.Model == "B4240" && d.ReaderAdapter != "bluefield-card-v1" { + return DeviceOutput{}, fmt.Errorf("device %s B4240 identity is outside the supported Redfish reader", d.RecordKey) + } + for _, key := range sortedObservationKeys(d.Observations) { + o := d.Observations[key] + switch key { + case "psid": + if o.State == "observed" { + if o.ReasonCode != "PSID_OBSERVED" || o.Tool != "redfish" { + return DeviceOutput{}, fmt.Errorf("device %s has inconsistent Redfish PSID evidence", d.RecordKey) + } + value, e := observedValue(o, "psid") + if e != nil { + return DeviceOutput{}, e + } + w.PSID = value + } else { + addIssue(&w.Issues, "psid", o) + } + case "bmc-administrator": + w.BMCAdministratorVerified = o.State == "observed" && o.ReasonCode == "BMC_ADMIN_VERIFIED" + if !w.BMCAdministratorVerified { + addIssue(&w.Issues, "bmcAdministratorVerified", o) + } + case "firmware-inventory": + if role == "dpus" { + w.FirmwareInventoryComplete = o.State == "observed" + addIssue(&w.Issues, "firmwareInventoryComplete", o) + } + case "managers", "bmc-time", "bmc-ntp", "client-certificate-support": + // These reads have no public fact consumer. + default: + if !strings.HasPrefix(key, "firmware.") { + addIssue(&w.Issues, key, o) + } + } + } + if d.PSID != "" { + if w.PSID != "" && w.PSID != d.PSID || hasIssue(w.Issues, "psid") { + return DeviceOutput{}, fmt.Errorf("device %s has conflicting PSID evidence", d.RecordKey) + } + w.PSID = d.PSID + } + if d.Authentication.BMCAdministrator != "" { + if d.Authentication.BMCAdministrator != "verified" || hasIssue(w.Issues, "bmcAdministratorVerified") { + return DeviceOutput{}, fmt.Errorf("device %s has conflicting BMC administrator evidence", d.RecordKey) + } + w.BMCAdministratorVerified = true + } + if d.FirmwareInventoryComplete { + if hasIssue(w.Issues, "firmwareInventoryComplete") { + return DeviceOutput{}, fmt.Errorf("device %s has conflicting firmware inventory completeness", d.RecordKey) + } + w.FirmwareInventoryComplete = role == "dpus" + } + // Positive source claims cannot outlive the identity/source recheck. + if w.ID == "" { + w.PSID, w.BMCAdministratorVerified, w.FirmwareInventoryComplete = "", false, false + } + if w.PSID != "" && (d.ReaderAdapter != "bluefield-card-v1" || w.Model != "B4240") { + return DeviceOutput{}, fmt.Errorf("device %s PSID lacks supported identity-bound Redfish evidence", d.RecordKey) + } + if role == "dpus" { + w.Firmware, err = firmwareToOutput(d.Firmware, d.Observations, &w.Issues) + if err != nil { + return DeviceOutput{}, err + } + } + if d.Runtime != nil { + w.Runtime, err = runtimeToOutput(d.Runtime, role, &w.Issues) + if err != nil { + return DeviceOutput{}, err + } + } + for i, a := range d.Adapters { + serial := a.SerialNumber + if serial == "" && a.IdentityState != "known" { + serial = a.SerialNumberRaw + } + w.Adapters = append(w.Adapters, AdapterOutput{Key: a.Key, SerialNumber: serial, Model: a.Model, Slot: a.Slot, + PCIAddress: a.PCIAddress, VendorID: a.VendorID, DeviceID: a.DeviceID, PCIFunctions: cloneFunctions(a.PCIFunctions)}) + if a.IdentityState != "known" { + code := a.ReasonCode + if code == "" { + code = "CARD_SERIAL_UNVERIFIED" + } + w.Issues = append(w.Issues, DeviceIssue{Field: fmt.Sprintf("adapters[%d].identity", i), Code: code}) + } + } + canonicalIssues(w.Issues) + if _, err = DeviceFromOutput(w, role); err != nil { + return DeviceOutput{}, err + } + return w, nil +} + +// DeviceFromOutput restores private facts under the v3 contract. Legacy objects +// must first pass their own versioned decoder; absence never upgrades them here. +func DeviceFromOutput(w DeviceOutput, role string) (Device, error) { + role, err := deviceRole(role) + if err != nil { + return Device{}, err + } + if err = validateDeviceOutput(w, role); err != nil { + return Device{}, err + } + d := Device{RecordKey: w.RecordKey, InputRef: w.InputRef, ID: w.ID, BMCEndpoint: w.BMCEndpoint, SSHEndpoint: w.SSHEndpoint, + SystemURI: w.SystemURI, SerialNumber: w.SerialNumber, SerialNumberRaw: w.SerialNumber, SystemUUID: w.SystemUUID, + Manufacturer: w.Manufacturer, Model: w.Model, AdvertisedResetTypes: slices.Clone(w.AdvertisedResetTypes), + ObservationState: "unknown", Observations: map[string]Observation{}} + if w.ID != "" { + d.ObservationState = "observed" + if role == "hosts" && w.SystemUUID != "" { + d.IdentityKey = []string{"system-uuid", strings.ToLower(w.SystemUUID)} + } else if role == "dpus" && w.SerialNumber != "" { + d.IdentityKey = []string{"nvidia", w.SerialNumber} + } + } + switch w.Method { + case "ssh": + d.ReaderAdapter = "linux-host-ssh-v1" + if role == "dpus" { + d.ReaderAdapter = "bluefield-arm-ssh-v1" + } + case "redfish": + if role == "dpus" { + d.ReaderAdapter = "bluefield-card-v1" + } + case "host-inventory": + d.ReaderAdapter = "host-card-v1" + } + if w.PSID != "" { + d.Observations["psid"] = Observation{State: "observed", Values: []string{w.PSID}, ReasonCode: "PSID_OBSERVED", Tool: "redfish"} + } + if w.BMCAdministratorVerified { + d.Observations["bmc-administrator"] = Observation{State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED", Tool: "redfish"} + } + if w.FirmwareInventoryComplete { + d.Observations["firmware-inventory"] = Observation{State: "observed"} + } + d.Firmware, err = firmwareFromOutput(w.Firmware, w.Issues) + if err != nil { + return Device{}, err + } + if w.Runtime != nil { + d.Runtime, err = runtimeFromOutput(w.Runtime, role, w.Issues) + if err != nil { + return Device{}, err + } + } + for i, a := range w.Adapters { + adapter := Adapter{Key: a.Key, SerialNumber: a.SerialNumber, SerialNumberRaw: a.SerialNumber, Model: a.Model, Slot: a.Slot, + PCIAddress: a.PCIAddress, VendorID: a.VendorID, DeviceID: a.DeviceID, PCIFunctions: cloneFunctions(a.PCIFunctions), Selection: "not-selected", IdentityState: "unknown"} + field := fmt.Sprintf("adapters[%d].identity", i) + if a.SerialNumber != "" && !hasIssue(w.Issues, field) { + adapter.IdentityState = "known" + } else { + // Correlation and HostCards consume SerialNumber directly. Preserve + // a failed candidate only as context, never as association authority. + adapter.SerialNumber = "" + } + for _, issue := range w.Issues { + if issue.Field == field { + adapter.ReasonCode = issue.Code + } + } + d.Adapters = append(d.Adapters, adapter) + } + for _, issue := range w.Issues { + switch issue.Field { + case "identity": + d.ReasonCode, d.Message = issue.Code, issue.Message + case "psid": + d.Observations["psid"] = issueObservation(issue) + case "bmcAdministratorVerified": + d.Observations["bmc-administrator"] = issueObservation(issue) + case "firmwareInventoryComplete": + d.Observations["firmware-inventory"] = issueObservation(issue) + default: + if strings.HasPrefix(issue.Field, "firmware.") { + d.Observations[issue.Field] = issueObservation(issue) + } + if !strings.HasPrefix(issue.Field, "runtime.") && !strings.HasPrefix(issue.Field, "adapters[") && !strings.HasPrefix(issue.Field, "firmware.") { + d.Observations[issue.Field] = issueObservation(issue) + } + } + } + return d, nil +} + +func validateDeviceOutput(w DeviceOutput, role string) error { + if err := validateHardwareIssueFields(w); err != nil { + return err + } + securityIssues := map[string]bool{} + for _, issue := range w.Issues { + if issue.Field == "" || issue.Code == "" || !safeFact(issue.Field) || !safeFact(issue.Code) { + return fmt.Errorf("device %s has an invalid issue", w.RecordKey) + } + if issue.Field == "identity" || issue.Field == "psid" || issue.Field == "bmcAdministratorVerified" || issue.Field == "firmwareInventoryComplete" { + if securityIssues[issue.Field] { + return fmt.Errorf("device %s has ambiguous %s issues", w.RecordKey, issue.Field) + } + securityIssues[issue.Field] = true + } + } + if w.ID != "" && hasIssue(w.Issues, "identity") { + return fmt.Errorf("device %s publishes identity with a failed recheck", w.RecordKey) + } + switch w.Method { + case "redfish": + if w.BMCEndpoint == "" || w.SSHEndpoint != "" { + return fmt.Errorf("device %s Redfish method requires only a BMC endpoint", w.RecordKey) + } + case "ssh": + if w.SSHEndpoint == "" { + return fmt.Errorf("device %s SSH method requires an SSH endpoint", w.RecordKey) + } + case "host-inventory": + if role != "dpus" || w.BMCEndpoint != "" || w.SSHEndpoint != "" { + return fmt.Errorf("device %s host inventory cannot assert an endpoint", w.RecordKey) + } + case "": + if w.ID != "" || w.BMCEndpoint != "" || w.SSHEndpoint != "" { + return fmt.Errorf("device %s requires its discovery method", w.RecordKey) + } + default: + return fmt.Errorf("device %s has unsupported method %q", w.RecordKey, w.Method) + } + if w.PSID != "" && (role != "dpus" || w.ID == "" || w.Method != "redfish" || w.Model != "B4240" || !validPSID(w.PSID) || hasIssue(w.Issues, "psid")) { + return fmt.Errorf("device %s has inconsistent verified Redfish PSID", w.RecordKey) + } + if w.BMCAdministratorVerified && (role != "dpus" || w.ID == "" || w.Method != "redfish" || hasIssue(w.Issues, "bmcAdministratorVerified")) { + return fmt.Errorf("device %s has inconsistent BMC administrator verification", w.RecordKey) + } + if (w.Firmware != nil || w.FirmwareInventoryComplete) && (role != "dpus" || w.Method != "redfish") { + return fmt.Errorf("device %s firmware inventory requires a DPU Redfish source", w.RecordKey) + } + if w.FirmwareInventoryComplete && (w.ID == "" || hasIssue(w.Issues, "firmwareInventoryComplete")) { + return fmt.Errorf("device %s has inconsistent firmware inventory completeness", w.RecordKey) + } + return nil +} + +func deviceRole(role string) (string, error) { + switch role { + case "host", "hosts": + return "hosts", nil + case "dpu", "dpus": + return "dpus", nil + default: + return "", fmt.Errorf("unknown inventory device role %q", role) + } +} + +func sortedObservationKeys(observations map[string]Observation) []string { + keys := make([]string, 0, len(observations)) + for key := range observations { + keys = append(keys, key) + } + sort.Strings(keys) + return keys +} + +func addIssue(issues *[]DeviceIssue, field string, observation Observation) { + if observation.State == "observed" || observation.ReasonCode == "NOT_APPLICABLE" || observation.State == "" && observation.ReasonCode == "" { + return + } + code := observation.ReasonCode + if code == "" { + code = "FACT_UNAVAILABLE" + } + issue := DeviceIssue{Field: field, Code: code, Message: observation.Message, Tool: observation.Tool} + if !slices.Contains(*issues, issue) { + *issues = append(*issues, issue) + } +} + +func hasIssue(issues []DeviceIssue, field string) bool { + return slices.ContainsFunc(issues, func(issue DeviceIssue) bool { return issue.Field == field }) +} + +func canonicalIssues(issues []DeviceIssue) { + slices.SortFunc(issues, func(a, b DeviceIssue) int { + if c := strings.Compare(a.Field, b.Field); c != 0 { + return c + } + if c := strings.Compare(a.Code, b.Code); c != 0 { + return c + } + if c := strings.Compare(a.Message, b.Message); c != 0 { + return c + } + return strings.Compare(a.Tool, b.Tool) + }) +} + +func issueObservation(issue DeviceIssue) Observation { + return Observation{State: "unavailable", ReasonCode: issue.Code, Message: issue.Message, Tool: issue.Tool} +} + +func observedValue(o Observation, field string) (string, error) { + if o.State != "observed" { + return "", nil + } + if len(o.Values) != 1 || !safeFact(o.Values[0]) { + return "", fmt.Errorf("%s requires exactly one nonempty measured value", field) + } + return o.Values[0], nil +} + +func safeFact(value string) bool { + return value != "" && len(value) <= 1024 && strings.TrimSpace(value) == value && !strings.ContainsAny(value, "\x00\r\n") +} + +func validPSID(value string) bool { + return len(value) <= 32 && strings.HasPrefix(value, "MT_") && strings.IndexFunc(value, func(r rune) bool { + allowed := r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '_' || r == '.' || r == '-' + return !allowed + }) < 0 +} + +func cloneFunctions(functions []PCIFunction) []PCIFunction { + out := slices.Clone(functions) + for i := range out { + out[i].Netdevs = slices.Clone(out[i].Netdevs) + } + return out +} diff --git a/pkg/dpf/inventory/device_output_contract_test.go b/pkg/dpf/inventory/device_output_contract_test.go new file mode 100644 index 00000000..792365e7 --- /dev/null +++ b/pkg/dpf/inventory/device_output_contract_test.go @@ -0,0 +1,174 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory_test + +import ( + "encoding/json" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" + "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func firmwareDevice() inventory.Device { + return inventory.Device{ID: "dpu-one", RecordKey: "dpu-source", InputRef: "/discovery/dpus/0", BMCEndpoint: "https://dpu.invalid", SerialNumber: "CARDONE", Model: "B3210", ObservationState: "observed", Observations: map[string]inventory.Observation{"firmware-inventory": {State: "observed"}}} +} + +func nicCheck(d inventory.Device) inventory.Check { + release, err := releases.Resolve("26.4.0") + Expect(err).NotTo(HaveOccurred()) + for _, check := range discovery.SoftwareChecks(&inventory.Inventory{Software: &release, DPUs: []inventory.Device{d}}) { + if check.ID == "dpf.software.nic" { + return check + } + } + ginkgo.Fail("NIC software check was not evaluated") + return inventory.Check{} +} + +var _ = ginkgo.Describe("typed firmware compatibility boundary", func() { + ginkgo.It("round trips reconciled whitespace variants without normalizing measured versions", func() { + d := firmwareDevice() + d.Firmware = []inventory.Firmware{ + {Component: "bmc", State: "known", RawVersion: "1.0", Activation: "unknown", ActivationReasonCode: "ACTIVATION_NOT_EXPOSED"}, + {Component: "bmc", State: "known", RawVersion: " 1.0 ", Activation: "running"}, + } + in := inventory.Inventory{FormatVersion: inventory.FormatVersion, DPUs: []inventory.Device{d}} + encoded, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + + var restored inventory.Inventory + Expect(json.Unmarshal(encoded, &restored)).To(Succeed()) + Expect(restored.DPUs[0].Firmware).To(ContainElements( + And(HaveField("RawVersion", "1.0"), HaveField("Activation", "unknown")), + And(HaveField("RawVersion", " 1.0 "), HaveField("Activation", "running")), + )) + check := nicCheck(inventory.Device{ID: "dpu-one", RecordKey: "dpu-source", SerialNumber: "CARDONE", Model: "B3210", ObservationState: "observed", + Observations: map[string]inventory.Observation{"firmware-inventory": {State: "observed"}}, + Firmware: []inventory.Firmware{{Component: "nic", State: "known", RawVersion: " 32.38.1002 ", Activation: "running"}}, + }) + Expect(check.State).To(Equal("unknown")) + Expect(check.ReasonCode).To(Equal("SOFTWARE_VERSION_UNCOMPARABLE")) + }) + + ginkgo.It("keeps actual conflicts and staged versions distinct with whitespace variants", func() { + conflict := firmwareDevice() + conflict.Firmware = []inventory.Firmware{ + {Component: "bmc", State: "conflict", ReasonCode: "FIRMWARE_VERSION_CONFLICT", RawVersion: "1.0", Activation: "running"}, + {Component: "bmc", State: "conflict", ReasonCode: "FIRMWARE_VERSION_CONFLICT", RawVersion: " 1.1 ", Activation: "unknown"}, + } + wire, err := inventory.DeviceToOutput(conflict, "dpus") + Expect(err).NotTo(HaveOccurred()) + restoredConflict, err := inventory.DeviceFromOutput(wire, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(restoredConflict.Firmware).To(HaveLen(2)) + for _, fact := range restoredConflict.Firmware { + Expect(fact.State).To(Equal("conflict")) + } + + staged := firmwareDevice() + staged.Firmware = []inventory.Firmware{ + {Component: "bmc", State: "known", RawVersion: "1.0", Activation: "running"}, + {Component: "bmc", State: "known", RawVersion: " 1.1 ", Activation: "staged"}, + } + stagedWire, err := inventory.DeviceToOutput(staged, "dpus") + Expect(err).NotTo(HaveOccurred()) + stagedRestored, err := inventory.DeviceFromOutput(stagedWire, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(stagedRestored.Firmware).To(ContainElements( + And(HaveField("RawVersion", "1.0"), HaveField("Activation", "running")), + And(HaveField("RawVersion", " 1.1 "), HaveField("Activation", "staged")), + )) + }) + + ginkgo.It("preserves padded measured versions without promoting compatibility", func() { + d := firmwareDevice() + d.Firmware = []inventory.Firmware{{Component: "nic", State: "known", RawVersion: " 32.38.1002 ", Activation: "running"}} + before := nicCheck(d) + Expect(before.State).To(Equal("unknown")) + Expect(before.ReasonCode).To(Equal("SOFTWARE_VERSION_UNCOMPARABLE")) + in := inventory.Inventory{FormatVersion: inventory.FormatVersion, DPUs: []inventory.Device{d}} + encoded, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + var restored inventory.Inventory + Expect(json.Unmarshal(encoded, &restored)).To(Succeed()) + Expect(restored.DPUs[0].Firmware[0].RawVersion).To(Equal(" 32.38.1002 ")) + Expect(nicCheck(restored.DPUs[0]).State).To(Equal(before.State)) + Expect(nicCheck(restored.DPUs[0]).ReasonCode).To(Equal(before.ReasonCode)) + }) + + ginkgo.DescribeTable("keeps unusable optional versions diagnostic without aborting publication", func(raw string) { + d := firmwareDevice() + d.Firmware = []inventory.Firmware{ + {Component: "nic", State: "known", RawVersion: "32.38.1002", Activation: "running"}, + {Component: "nic", State: "unknown", RawVersion: raw, Activation: "unknown", ActivationReasonCode: "CROSS_ORIGIN_LINK", ReasonCode: "VERSION_NOT_EXPOSED"}, + } + w, err := inventory.DeviceToOutput(d, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(w.Firmware.NIC).To(HaveLen(1)) + Expect(w.Issues).To(ContainElement(inventory.DeviceIssue{Field: "firmware.nic", Code: "VERSION_INVALID", Message: "The firmware version is blank or contains unsupported characters."})) + restored, err := inventory.DeviceFromOutput(w, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(nicCheck(restored).State).To(Equal("passed")) + again, err := inventory.DeviceToOutput(restored, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(again).To(Equal(w)) + _, err = json.Marshal(inventory.Inventory{FormatVersion: inventory.FormatVersion, DPUs: []inventory.Device{d}}) + Expect(err).NotTo(HaveOccurred()) + }, ginkgo.Entry("blank whitespace", " "), ginkgo.Entry("newline", "v1.2\n"), ginkgo.Entry("control character", "v1.2\x00")) + + ginkgo.It("retains actual conflicts when an optional version is unusable", func() { + d := firmwareDevice() + d.Firmware = []inventory.Firmware{ + {Component: "nic", State: "known", RawVersion: "32.38.1002", Activation: "running"}, + {Component: "nic", State: "unknown", RawVersion: " ", Activation: "unknown", ActivationReasonCode: "ACTIVATION_CONFLICT"}, + } + w, err := inventory.DeviceToOutput(d, "dpus") + Expect(err).NotTo(HaveOccurred()) + restored, err := inventory.DeviceFromOutput(w, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(nicCheck(restored).State).To(Equal("unknown")) + Expect(nicCheck(restored).ReasonCode).To(Equal("SOFTWARE_VERSION_CONFLICT")) + }) + + ginkgo.It("never clears a conflict veto with a later per-entry diagnostic", func() { + for _, activation := range []bool{false, true} { + for _, reverse := range []bool{false, true} { + d := firmwareDevice() + d.Firmware = []inventory.Firmware{{Component: "nic", State: "known", RawVersion: "32.38.1002", Activation: "unknown"}} + d.Runtime = &inventory.RuntimeObservation{Hardware: &inventory.HardwareObservation{Cards: []inventory.CardHardware{{Domain: "arm-pci", PCIAddress: "0000:21:00.0", SerialNumber: d.SerialNumber, Firmware: inventory.Observation{State: "observed", Values: []string{"32.38.1002"}}}}}} + w, err := inventory.DeviceToOutput(d, "dpus") + Expect(err).NotTo(HaveOccurred()) + w.Issues = []inventory.DeviceIssue{{Field: "firmware.nic", Code: "FIRMWARE_VERSION_CONFLICT"}, {Field: "firmware.nic[0]", Code: "VERSION_INVALID"}} + if activation { + w.Issues = []inventory.DeviceIssue{{Field: "firmware.nic.activation", Code: "ACTIVATION_CONFLICT"}, {Field: "firmware.nic[0].activation", Code: "CROSS_ORIGIN_LINK"}} + } + if reverse { + w.Issues[0], w.Issues[1] = w.Issues[1], w.Issues[0] + } + restored, err := inventory.DeviceFromOutput(w, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(nicCheck(restored).ReasonCode).To(Equal("SOFTWARE_VERSION_CONFLICT")) + } + } + }) + + ginkgo.DescribeTable("rejects failure locations that would otherwise be ignored", func(field, code string) { + w := inventory.DeviceOutput{ID: "dpu-one", RecordKey: "dpu-source", Method: "redfish", BMCEndpoint: "https://dpu.invalid", SerialNumber: "CARDONE", Model: "B3210", FirmwareInventoryComplete: true, + Firmware: &inventory.FirmwareOutput{NIC: []inventory.FirmwareVersionOutput{{Version: "32.38.1002", Activation: "running"}}}, + Issues: []inventory.DeviceIssue{{Field: field, Code: code}}} + _, err := inventory.DeviceFromOutput(w, "dpus") + Expect(err).To(HaveOccurred()) + }, + ginkgo.Entry("mis-scoped activation conflict", "firmware.nic", "ACTIVATION_CONFLICT"), + ginkgo.Entry("absent firmware index", "firmware.nic[9]", "ACTIVATION_CONFLICT"), + ginkgo.Entry("unknown component", "firmware.unknown[0].activation", "ACTIVATION_CONFLICT"), + ginkgo.Entry("unsupported firmware property", "firmware.nic[0].unrecognized", "ACTIVATION_CONFLICT"), + ginkgo.Entry("noncanonical index", "firmware.nic[00].activation", "ACTIVATION_CONFLICT"), + ginkgo.Entry("absent adapter index", "adapters[9].identity", "CARD_SERIAL_UNVERIFIED"), + ginkgo.Entry("absent hardware card", "runtime.hardware.cards[9].psid", "PSID_CONFLICT"), + ) +}) diff --git a/pkg/dpf/inventory/device_output_firmware.go b/pkg/dpf/inventory/device_output_firmware.go new file mode 100644 index 00000000..9689c445 --- /dev/null +++ b/pkg/dpf/inventory/device_output_firmware.go @@ -0,0 +1,198 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "fmt" + "slices" + "strings" +) + +type firmwareOutputSlot struct { + component string + versions *[]FirmwareVersionOutput +} + +func firmwareOutputSlots(w *FirmwareOutput) []firmwareOutputSlot { + return []firmwareOutputSlot{ + {"bmc", &w.BMC}, {"nic", &w.NIC}, {"cec", &w.CEC}, {"atf", &w.ATF}, {"uefi", &w.UEFI}, {"atf-uefi", &w.ATFUEFI}, + {"dpu-bsp", &w.BSP}, {"dpu-os", &w.OS}, {"golden", &w.Golden}, {"bmc-erot", &w.BMCERoT}, {"cpu-erot", &w.CPUERoT}, + } +} + +func firmwareToOutput(firmware []Firmware, observations map[string]Observation, issues *[]DeviceIssue) (*FirmwareOutput, error) { + w := &FirmwareOutput{} + used := false + for _, slot := range firmwareOutputSlots(w) { + field := "firmware." + slot.component + addFirmwareIssue(issues, field, observations[field]) + addFirmwareIssue(issues, field+".activation", observations[field+".activation"]) + activationConflict := observations[field+".activation"].ReasonCode == "ACTIVATION_CONFLICT" + for _, f := range firmware { + activationConflict = activationConflict || f.Component == slot.component && !usableFirmwareVersion(f.RawVersion) && f.ActivationReasonCode == "ACTIVATION_CONFLICT" + } + for _, f := range firmware { + if f.Component != slot.component { + continue + } + usable := usableFirmwareVersion(f.RawVersion) + if !usable && (f.State != "known" || f.ReasonCode != "") { + code := f.ReasonCode + if f.State == "conflict" { + code = "FIRMWARE_VERSION_CONFLICT" + } + addFirmwareIssue(issues, field, Observation{State: "unknown", ReasonCode: code, Message: f.Message}) + } + if !usable && f.ActivationReasonCode != "" { + addFirmwareIssue(issues, field+".activation", Observation{State: "unknown", ReasonCode: f.ActivationReasonCode, Message: f.ActivationMessage}) + } + if !usable { + if f.RawVersion != "" || f.State == "known" { + addFirmwareIssue(issues, field, Observation{State: "unknown", ReasonCode: "VERSION_INVALID", Message: "The firmware version is blank or contains unsupported characters."}) + } + continue + } + activation := f.Activation + if activation == "unknown" || activationConflict || f.ActivationReasonCode == "ACTIVATION_CONFLICT" { + activation = "" + } + if activation != "" && activation != "running" && activation != "staged" { + return nil, fmt.Errorf("%s has unsupported firmware activation", field) + } + entry := FirmwareVersionOutput{Version: f.RawVersion, Activation: activation} + if !slices.Contains(*slot.versions, entry) { + *slot.versions = append(*slot.versions, entry) + used = true + } + } + slices.SortFunc(*slot.versions, func(a, b FirmwareVersionOutput) int { + if c := strings.Compare(a.Activation, b.Activation); c != 0 { + return c + } + return strings.Compare(a.Version, b.Version) + }) + for _, f := range firmware { + if f.Component != slot.component || !usableFirmwareVersion(f.RawVersion) { + continue + } + activation := f.Activation + if activation == "unknown" || activationConflict || f.ActivationReasonCode == "ACTIVATION_CONFLICT" { + activation = "" + } + index := slices.Index(*slot.versions, FirmwareVersionOutput{Version: f.RawVersion, Activation: activation}) + entryField := fmt.Sprintf("%s[%d]", field, index) + if f.State == "conflict" || f.ReasonCode == "FIRMWARE_VERSION_CONFLICT" { + addFirmwareIssue(issues, field, Observation{State: "unknown", ReasonCode: "FIRMWARE_VERSION_CONFLICT", Message: f.Message}) + } else if f.State != "known" || f.ReasonCode != "" { + addFirmwareIssue(issues, entryField, Observation{State: "unknown", ReasonCode: f.ReasonCode, Message: f.Message}) + } + if f.ActivationReasonCode != "" { + addFirmwareIssue(issues, entryField+".activation", Observation{State: "unknown", ReasonCode: f.ActivationReasonCode, Message: f.ActivationMessage}) + } + } + } + if !used { + return nil, nil + } + return w, nil +} + +func firmwareFromOutput(w *FirmwareOutput, issues []DeviceIssue) ([]Firmware, error) { + if w == nil { + w = &FirmwareOutput{} + } + if err := validateFirmwareIssues(w, issues); err != nil { + return nil, err + } + var firmware []Firmware + for _, slot := range firmwareOutputSlots(w) { + field := "firmware." + slot.component + nonstaged := map[string]bool{} + conflict := false + for _, issue := range issues { + if issue.Field == field && issue.Code == "FIRMWARE_VERSION_CONFLICT" { + conflict = true + } + } + for index, version := range *slot.versions { + if !usableFirmwareVersion(version.Version) || version.Activation != "" && version.Activation != "running" && version.Activation != "staged" { + return nil, fmt.Errorf("%s requires a measured version and valid activation", field) + } + f := Firmware{Component: slot.component, State: "known", RawVersion: version.Version, Activation: version.Activation} + if f.Activation == "" { + f.Activation = "unknown" + } + if version.Activation != "staged" { + // Match Redfish reconciliation: whitespace variants are the same + // version for conflict detection, while the public fact retains the + // exact measured spelling for conservative compatibility checks. + nonstaged[strings.TrimSpace(version.Version)] = true + } + for _, issue := range issues { + entryField := fmt.Sprintf("%s[%d]", field, index) + if issue.Field == entryField { + f.State, f.ReasonCode, f.Message = "unknown", issue.Code, issue.Message + if issue.Code == "FIRMWARE_VERSION_CONFLICT" { + f.State = "conflict" + } + } + if issue.Field == entryField+".activation" { + if version.Activation != "" { + return nil, fmt.Errorf("%s publishes activation with a failed activation observation", field) + } + f.ActivationReasonCode, f.ActivationMessage = issue.Code, issue.Message + } + } + // Component conflicts veto compatibility regardless of issue order + // or an additional per-entry read failure. + if conflict { + f.State, f.ReasonCode = "conflict", "FIRMWARE_VERSION_CONFLICT" + } + for _, issue := range issues { + if (issue.Field == field+".activation" || issue.Field == fmt.Sprintf("%s[%d].activation", field, index)) && issue.Code == "ACTIVATION_CONFLICT" { + f.Activation, f.ActivationReasonCode, f.ActivationMessage = "unknown", issue.Code, issue.Message + } + } + firmware = append(firmware, f) + } + if len(nonstaged) > 1 && !conflict { + return nil, fmt.Errorf("%s has conflicting current versions without a conflict issue", field) + } + if len(*slot.versions) == 0 { + for _, issue := range issues { + if issue.Field == field { + state := "unknown" + if issue.Code == "FIRMWARE_VERSION_CONFLICT" { + state = "conflict" + } + firmware = append(firmware, Firmware{Component: slot.component, State: state, Activation: "unknown", ReasonCode: issue.Code, Message: issue.Message}) + } + if issue.Field == field+".activation" && issue.Code == "ACTIVATION_CONFLICT" { + firmware = append(firmware, Firmware{Component: slot.component, State: "unknown", ReasonCode: "VERSION_UNAVAILABLE", Activation: "unknown", ActivationReasonCode: issue.Code, ActivationMessage: issue.Message}) + } + } + } + } + return firmware, nil +} + +// Raw versions are compared without normalization by software checks. Retain +// safe surrounding whitespace: trimming could turn an uncomparable value into +// compatibility evidence. Unusable optional measurements remain diagnostics. +func usableFirmwareVersion(version string) bool { + return len(version) <= 1024 && strings.TrimSpace(version) != "" && !strings.ContainsFunc(version, func(r rune) bool { + return r < ' ' || r == 0x7f + }) +} + +// Routine absent/unknown component properties are represented by absent typed +// facts. Software checks derive the missing measurement; only actual failures +// and mismatches warrant a persisted issue. +func addFirmwareIssue(issues *[]DeviceIssue, field string, observation Observation) { + switch observation.ReasonCode { + case "COMPONENT_NOT_EXPOSED", "VERSION_UNAVAILABLE", "VERSION_NOT_EXPOSED", "ACTIVATION_NOT_EXPOSED": + return + } + addIssue(issues, field, observation) +} diff --git a/pkg/dpf/inventory/device_output_hardware.go b/pkg/dpf/inventory/device_output_hardware.go new file mode 100644 index 00000000..c821bbda --- /dev/null +++ b/pkg/dpf/inventory/device_output_hardware.go @@ -0,0 +1,388 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "fmt" + "regexp" + "slices" + "strconv" + "strings" +) + +var outputToken = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._()+-]{0,127}$`) + +func runtimeToOutput(r *RuntimeObservation, role string, issues *[]DeviceIssue) (*RuntimeOutput, error) { + w := &RuntimeOutput{Hostname: r.Hostname, OS: r.OS, Interfaces: cloneInterfaces(r.Interfaces), Routes: slices.Clone(r.Routes), DNS: slices.Clone(r.DNS), Network: cloneNetwork(r.Network)} + for _, key := range sortedObservationKeys(r.Observations) { + o := r.Observations[key] + switch key { + case "board-product", "mft-version": + field := "runtime.boardProduct" + if key == "mft-version" { + field = "runtime.mftVersion" + } + value, err := observedValue(o, field) + if err != nil { + return nil, err + } + if key == "board-product" { + w.BoardProduct = value + } else { + w.MFTVersion = value + } + addIssue(issues, field, o) + case "pciScope", "bmcAddress": + // PCI scope is constant prose; BMC address already has a field. + if key == "bmcAddress" { + addIssue(issues, "bmcEndpoint", o) + } + default: + addIssue(issues, "runtime."+key, o) + } + } + if r.Hardware == nil { + return w, nil + } + w.Hardware = &HardwareOutput{} + if role == "hosts" { + if r.Hardware.RShim.State == "observed" { + service, err := rshimToOutput(r.Hardware.RShim.Values) + if err != nil { + return nil, err + } + w.Hardware.RShim = service + } else { + addIssue(issues, "runtime.hardware.rshim", r.Hardware.RShim) + } + } + for index, card := range r.Hardware.Cards { + prefix := fmt.Sprintf("runtime.hardware.cards[%d].", index) + out := CardOutput{PCIAddress: card.PCIAddress, Domain: card.Domain, Model: card.Model, SerialNumber: card.SerialNumber} + var err error + out.FirmwareVersion, err = observedValue(card.Firmware, prefix+"firmwareVersion") + if err != nil { + return nil, err + } + out.PSID, err = observedValue(card.PSID, prefix+"psid") + if err != nil { + return nil, err + } + addIssue(issues, prefix+"firmwareVersion", card.Firmware) + addIssue(issues, prefix+"psid", card.PSID) + for _, parameter := range []struct{ key, field string }{ + {"INTERNAL_CPU_OFFLOAD_ENGINE", "internalCPUOffloadEngine"}, {"INTERNAL_CPU_RSHIM", "internalCPURShim"}, {"NUM_OF_VFS", "numVFs"}, + } { + o := card.NVConfig[parameter.key] + value, err := observedValue(o, prefix+"nextBoot."+parameter.field) + if err != nil { + return nil, err + } + addIssue(issues, prefix+"nextBoot."+parameter.field, o) + if value == "" { + continue + } + value, ok := strings.CutPrefix(value, "next-boot=") + if !ok || !outputToken.MatchString(value) { + return nil, fmt.Errorf("%s must be an explicit next-boot value", prefix+parameter.field) + } + if out.NextBoot == nil { + out.NextBoot = &NVConfigOutput{} + } + switch parameter.key { + case "INTERNAL_CPU_OFFLOAD_ENGINE": + out.NextBoot.InternalCPUOffloadEngine = value + case "INTERNAL_CPU_RSHIM": + out.NextBoot.InternalCPURShim = value + case "NUM_OF_VFS": + n, err := strconv.ParseUint(value, 10, 32) + if err != nil { + return nil, fmt.Errorf("%snumVFs is not a valid VF count", prefix) + } + count := uint32(n) + out.NextBoot.NumVFs = &count + } + } + if role == "dpus" { + if card.Privilege.State == "observed" { + out.HostPrivileges, err = privilegeToOutput(card.Privilege.Values) + if err != nil { + return nil, err + } + } else { + addIssue(issues, prefix+"hostPrivileges", card.Privilege) + } + } + w.Hardware.Cards = append(w.Hardware.Cards, out) + } + return w, nil +} + +func runtimeFromOutput(w *RuntimeOutput, role string, issues []DeviceIssue) (*RuntimeObservation, error) { + r := &RuntimeObservation{Hostname: w.Hostname, OS: w.OS, Interfaces: cloneInterfaces(w.Interfaces), Routes: slices.Clone(w.Routes), DNS: slices.Clone(w.DNS), + Network: cloneNetwork(w.Network), Observations: map[string]Observation{}} + for _, value := range []struct{ name, key, value, tool string }{ + {"boardProduct", "board-product", w.BoardProduct, "ipmitool"}, {"mftVersion", "mft-version", w.MFTVersion, "mlxfwmanager"}, + } { + if value.value != "" { + if !safeFact(value.value) || hasIssue(issues, "runtime."+value.name) { + return nil, fmt.Errorf("runtime.%s has conflicting or invalid values", value.name) + } + r.Observations[value.key] = Observation{State: "observed", Values: []string{value.value}, Tool: value.tool} + } + } + for _, issue := range issues { + if strings.HasPrefix(issue.Field, "runtime.") && !strings.HasPrefix(issue.Field, "runtime.hardware.") { + key := strings.TrimPrefix(issue.Field, "runtime.") + switch key { + case "boardProduct": + key = "board-product" + case "mftVersion": + key = "mft-version" + } + r.Observations[key] = issueObservation(issue) + } + } + if w.Hardware == nil { + return r, nil + } + r.Hardware = &HardwareObservation{} + if w.Hardware.RShim != nil { + if role != "hosts" || hasIssue(issues, "runtime.hardware.rshim") || !outputToken.MatchString(w.Hardware.RShim.LoadState) || !outputToken.MatchString(w.Hardware.RShim.ActiveState) { + return nil, fmt.Errorf("rshim requires valid host service states without a read failure") + } + r.Hardware.RShim = Observation{State: "observed", Values: []string{"LoadState=" + w.Hardware.RShim.LoadState, "ActiveState=" + w.Hardware.RShim.ActiveState}, Tool: "systemctl"} + } + for _, issue := range issues { + if issue.Field == "runtime.hardware.rshim" { + r.Hardware.RShim = issueObservation(issue) + } + } + seen := map[string]bool{} + for index, card := range w.Hardware.Cards { + prefix := fmt.Sprintf("runtime.hardware.cards[%d].", index) + if card.Domain != "host-pci" && card.Domain != "arm-pci" || card.PCIAddress == "" || seen[card.Domain+"/"+card.PCIAddress] { + return nil, fmt.Errorf("%srequires a unique PCI address and domain", prefix) + } + seen[card.Domain+"/"+card.PCIAddress] = true + out := CardHardware{PCIAddress: card.PCIAddress, Domain: card.Domain, Model: card.Model, SerialNumber: card.SerialNumber, NVConfig: map[string]Observation{}} + for _, value := range []struct { + name, value string + target *Observation + }{ + {"firmwareVersion", card.FirmwareVersion, &out.Firmware}, {"psid", card.PSID, &out.PSID}, + } { + if value.value != "" { + if !outputToken.MatchString(value.value) || hasIssue(issues, prefix+value.name) { + return nil, fmt.Errorf("%s%s has invalid or conflicting facts", prefix, value.name) + } + *value.target = Observation{State: "observed", Values: []string{value.value}, Tool: "mlxfwmanager"} + } + } + if card.NextBoot != nil { + for _, parameter := range []struct{ key, name, value string }{ + {"INTERNAL_CPU_OFFLOAD_ENGINE", "internalCPUOffloadEngine", card.NextBoot.InternalCPUOffloadEngine}, + {"INTERNAL_CPU_RSHIM", "internalCPURShim", card.NextBoot.InternalCPURShim}, + } { + if parameter.value != "" { + if !outputToken.MatchString(parameter.value) || hasIssue(issues, prefix+"nextBoot."+parameter.name) { + return nil, fmt.Errorf("%snextBoot.%s has invalid or conflicting facts", prefix, parameter.name) + } + out.NVConfig[parameter.key] = Observation{State: "observed", Values: []string{"next-boot=" + parameter.value}, Tool: "mlxconfig"} + } + } + if card.NextBoot.NumVFs != nil { + if hasIssue(issues, prefix+"nextBoot.numVFs") { + return nil, fmt.Errorf("%snextBoot.numVFs conflicts with a read failure", prefix) + } + out.NVConfig["NUM_OF_VFS"] = Observation{State: "observed", Values: []string{"next-boot=" + strconv.FormatUint(uint64(*card.NextBoot.NumVFs), 10)}, Tool: "mlxconfig"} + } + } + if len(card.HostPrivileges) != 0 { + if role != "dpus" || hasIssue(issues, prefix+"hostPrivileges") { + return nil, fmt.Errorf("%shostPrivileges has invalid source or conflicting facts", prefix) + } + values, err := privilegeFromOutput(card.HostPrivileges) + if err != nil { + return nil, err + } + out.Privilege = Observation{State: "observed", Values: values, Tool: "mlxprivhost"} + } + for _, issue := range issues { + switch issue.Field { + case prefix + "firmwareVersion": + out.Firmware = issueObservation(issue) + case prefix + "psid": + out.PSID = issueObservation(issue) + case prefix + "hostPrivileges": + out.Privilege = issueObservation(issue) + case prefix + "nextBoot.internalCPUOffloadEngine": + out.NVConfig["INTERNAL_CPU_OFFLOAD_ENGINE"] = issueObservation(issue) + case prefix + "nextBoot.internalCPURShim": + out.NVConfig["INTERNAL_CPU_RSHIM"] = issueObservation(issue) + case prefix + "nextBoot.numVFs": + out.NVConfig["NUM_OF_VFS"] = issueObservation(issue) + } + } + r.Hardware.Cards = append(r.Hardware.Cards, out) + } + return r, nil +} + +func rshimToOutput(values []string) (*RShimServiceOutput, error) { + w := &RShimServiceOutput{} + for _, value := range values { + key, value, ok := strings.Cut(value, "=") + if !ok || !outputToken.MatchString(value) { + return nil, fmt.Errorf("invalid rshim service value") + } + switch key { + case "LoadState": + if w.LoadState != "" { + return nil, fmt.Errorf("duplicate rshim LoadState") + } + w.LoadState = value + case "ActiveState": + if w.ActiveState != "" { + return nil, fmt.Errorf("duplicate rshim ActiveState") + } + w.ActiveState = value + default: + return nil, fmt.Errorf("unsupported rshim property %q", key) + } + } + if w.LoadState == "" || w.ActiveState == "" { + return nil, fmt.Errorf("rshim service requires load and active states") + } + return w, nil +} + +func privilegeToOutput(values []string) ([]HostPrivilegeOutput, error) { + byHost := map[uint32]*HostPrivilegeOutput{} + seen := map[string]bool{} + for _, value := range values { + key, value, ok := strings.Cut(value, "=") + host, property, hasProperty := strings.Cut(key, "].") + host, hasHost := strings.CutPrefix(host, "host[") + n, err := strconv.ParseUint(host, 10, 32) + if !ok || !hasProperty || !hasHost || err != nil || seen[key] { + return nil, fmt.Errorf("invalid or duplicate host privilege property") + } + seen[key] = true + index := uint32(n) + entry := byHost[index] + if entry == nil { + entry = &HostPrivilegeOutput{HostIndex: index} + byHost[index] = entry + } + if property == "level" { + entry.Level = value + continue + } + if value != "TRUE" && value != "FALSE" { + return nil, fmt.Errorf("invalid host privilege boolean") + } + flag := value == "TRUE" + switch property { + case "disable_rshim": + entry.DisableRShim = &flag + case "disable_tracer": + entry.DisableTracer = &flag + case "disable_port_owner": + entry.DisablePortOwner = &flag + case "disable_counter_rd": + entry.DisableCounterRead = &flag + default: + return nil, fmt.Errorf("unsupported host privilege property %q", property) + } + } + result := make([]HostPrivilegeOutput, 0, len(byHost)) + for _, entry := range byHost { + result = append(result, *entry) + } + slices.SortFunc(result, func(a, b HostPrivilegeOutput) int { + if a.HostIndex < b.HostIndex { + return -1 + } + if a.HostIndex > b.HostIndex { + return 1 + } + return 0 + }) + if _, err := privilegeFromOutput(result); err != nil { + return nil, err + } + return result, nil +} + +func privilegeFromOutput(values []HostPrivilegeOutput) ([]string, error) { + if len(values) == 0 { + return nil, fmt.Errorf("host privileges require at least one host level") + } + seen := map[uint32]bool{} + var result []string + for _, entry := range values { + if seen[entry.HostIndex] || entry.Level != "PRIVILEGED" && entry.Level != "RESTRICTED" { + return nil, fmt.Errorf("host privileges require unique host indices and valid levels") + } + seen[entry.HostIndex] = true + prefix := "host[" + strconv.FormatUint(uint64(entry.HostIndex), 10) + "]." + result = append(result, prefix+"level="+entry.Level) + for _, value := range []struct { + key string + flag *bool + }{ + {"disable_rshim", entry.DisableRShim}, {"disable_tracer", entry.DisableTracer}, {"disable_port_owner", entry.DisablePortOwner}, {"disable_counter_rd", entry.DisableCounterRead}, + } { + if value.flag != nil { + text := "FALSE" + if *value.flag { + text = "TRUE" + } + result = append(result, prefix+value.key+"="+text) + } + } + } + return result, nil +} + +func cloneInterfaces(in []Interface) []Interface { + out := slices.Clone(in) + for i := range out { + out[i].Addresses = slices.Clone(out[i].Addresses) + } + return out +} + +func cloneNetwork(in *NetworkObservation) *NetworkObservation { + if in == nil { + return nil + } + out := *in + out.Links = slices.Clone(in.Links) + out.LLDP = slices.Clone(in.LLDP) + out.Neighbors = slices.Clone(in.Neighbors) + for i := range out.Neighbors { + out.Neighbors[i].State = slices.Clone(out.Neighbors[i].State) + } + out.Resolver.Search = slices.Clone(in.Resolver.Search) + out.Resolver.DNSMasqServers = slices.Clone(in.Resolver.DNSMasqServers) + if in.Resolver.DNSMasqNoResolv != nil { + flag := *in.Resolver.DNSMasqNoResolv + out.Resolver.DNSMasqNoResolv = &flag + } + clone := func(in map[string][]string) map[string][]string { + if in == nil { + return nil + } + out := make(map[string][]string, len(in)) + for key, values := range in { + out[key] = slices.Clone(values) + } + return out + } + out.Resolver.PerLink = clone(in.Resolver.PerLink) + out.Resolver.Domains = clone(in.Resolver.Domains) + return &out +} diff --git a/pkg/dpf/inventory/device_output_issues.go b/pkg/dpf/inventory/device_output_issues.go new file mode 100644 index 00000000..f832d0f7 --- /dev/null +++ b/pkg/dpf/inventory/device_output_issues.go @@ -0,0 +1,88 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "fmt" + "regexp" + "strconv" + "strings" +) + +var firmwareIssueField = regexp.MustCompile(`^firmware\.([a-z][a-z0-9-]*)(?:\[([0-9]+)\])?(\.activation)?$`) +var adapterIssueField = regexp.MustCompile(`^adapters\[([0-9]+)\]\.identity$`) +var cardIssueField = regexp.MustCompile(`^runtime\.hardware\.cards\[([0-9]+)\]\.(firmwareVersion|psid|hostPrivileges|nextBoot\.(internalCPUOffloadEngine|internalCPURShim|numVFs))$`) + +func issueIndex(text string, length int) (int, bool) { + index, err := strconv.Atoi(text) + return index, err == nil && index >= 0 && index < length && strconv.Itoa(index) == text +} + +// A diagnostic must address the actual typed fact that its decoder consumes. +// Otherwise a misspelled scope or absent row could silently discard a failure. +func validateFirmwareIssues(w *FirmwareOutput, issues []DeviceIssue) error { + for _, issue := range issues { + if !strings.HasPrefix(issue.Field, "firmware.") { + continue + } + match := firmwareIssueField.FindStringSubmatch(issue.Field) + if match == nil { + return fmt.Errorf("invalid firmware issue field %q", issue.Field) + } + var versions *[]FirmwareVersionOutput + for _, slot := range firmwareOutputSlots(w) { + if slot.component == match[1] { + versions = slot.versions + break + } + } + if versions == nil { + return fmt.Errorf("firmware issue names an unsupported component %q", match[1]) + } + indexed, activation := match[2] != "", match[3] != "" + if indexed { + if _, valid := issueIndex(match[2], len(*versions)); !valid { + return fmt.Errorf("firmware issue names an absent version %q", issue.Field) + } + } + if issue.Code == "ACTIVATION_CONFLICT" && !activation { + return fmt.Errorf("activation conflict requires an activation field: %q", issue.Field) + } + if issue.Code == "FIRMWARE_VERSION_CONFLICT" && (indexed || activation) { + return fmt.Errorf("firmware version conflict requires its component field: %q", issue.Field) + } + } + return nil +} + +func validateHardwareIssueFields(w DeviceOutput) error { + for _, issue := range w.Issues { + if strings.HasPrefix(issue.Field, "adapters[") { + match := adapterIssueField.FindStringSubmatch(issue.Field) + if match == nil { + return fmt.Errorf("invalid adapter issue field %q", issue.Field) + } + if _, valid := issueIndex(match[1], len(w.Adapters)); !valid { + return fmt.Errorf("adapter issue names an absent adapter %q", issue.Field) + } + } + if !strings.HasPrefix(issue.Field, "runtime.hardware.") { + continue + } + if w.Runtime == nil || w.Runtime.Hardware == nil { + return fmt.Errorf("hardware issue has no hardware context: %q", issue.Field) + } + if issue.Field == "runtime.hardware.rshim" { + continue + } + match := cardIssueField.FindStringSubmatch(issue.Field) + if match == nil { + return fmt.Errorf("invalid hardware issue field %q", issue.Field) + } + if _, valid := issueIndex(match[1], len(w.Runtime.Hardware.Cards)); !valid { + return fmt.Errorf("hardware issue names an absent card %q", issue.Field) + } + } + return nil +} diff --git a/pkg/dpf/inventory/device_output_test.go b/pkg/dpf/inventory/device_output_test.go new file mode 100644 index 00000000..0840bee1 --- /dev/null +++ b/pkg/dpf/inventory/device_output_test.go @@ -0,0 +1,188 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "encoding/json" + "strings" + + "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +func outputDeviceFixture() Device { + return Device{ID: "dpu-one", RecordKey: "record-one", InputRef: "/discovery/dpus/0", ObservationState: "observed", + BMCEndpoint: "https://192.0.2.1", Model: "B4240", SerialNumber: "MT001234", ReaderAdapter: "bluefield-card-v1", + Observations: map[string]Observation{ + "psid": {State: "observed", Values: []string{"MT_000001"}, ReasonCode: "PSID_OBSERVED", Tool: "redfish"}, + "bmc-administrator": {State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED", Tool: "redfish"}, + "firmware-inventory": {State: "observed"}, + }, + } +} + +var _ = ginkgo.Describe("typed DeviceOutput", func() { + ginkgo.It("publishes named component versions and required authorization without bookkeeping", func() { + d := outputDeviceFixture() + d.Qualification, d.InventoryCompleteness = "unqualified", "partial" + d.Time = TimeObservation{State: "known", DateTime: "2026-11-13"} + d.Firmware = []Firmware{ + {Component: "bmc", RawVersion: "26.1", State: "known", Activation: "running", RawID: "private-id", RawName: "private-name", ResourceURI: "/private/resource", EvidenceRefs: []string{"private-evidence"}}, + {Component: "bmc", RawVersion: "26.2", State: "known", Activation: "staged"}, + {Component: "nic", RawVersion: "32.4", State: "known", Activation: "unknown", ActivationReasonCode: "ACTIVATION_NOT_EXPOSED"}, + {Component: "dpu-os", State: "unknown", ReasonCode: "COMPONENT_NOT_EXPOSED"}, + {Component: "other", RawVersion: "arbitrary", State: "known", Activation: "unknown"}, + } + w, err := DeviceToOutput(d, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(w.PSID).To(Equal("MT_000001")) + Expect(w.BMCAdministratorVerified).To(BeTrue()) + Expect(w.FirmwareInventoryComplete).To(BeTrue()) + Expect(w.Firmware.BMC).To(HaveLen(2)) + Expect(w.Firmware.NIC[0].Activation).To(BeEmpty()) + Expect(w.Issues).To(BeEmpty()) + encoded, err := json.Marshal(w) + Expect(err).NotTo(HaveOccurred()) + for _, removed := range []string{"observations", "observationState", "readerAdapter", "qualification", "inventoryCompleteness", "authentication", "evidenceRefs", "rawID", "rawName", "resourceURI", "private-", "COMPONENT_NOT_EXPOSED", "ACTIVATION_NOT_EXPOSED", "arbitrary"} { + Expect(string(encoded)).NotTo(ContainSubstring(removed)) + } + for _, marshal := range []func(any) ([]byte, error){json.Marshal, yaml.Marshal} { + data, err := marshal(w) + Expect(err).NotTo(HaveOccurred()) + var copy DeviceOutput + if strings.HasPrefix(string(data), "{") { + Expect(json.Unmarshal(data, ©)).To(Succeed()) + } else { + Expect(yaml.Unmarshal(data, ©)).To(Succeed()) + } + restored, err := DeviceFromOutput(copy, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(restored.ObservationState).To(Equal("observed")) + Expect(restored.ReaderAdapter).To(Equal("bluefield-card-v1")) + Expect(restored.Observations["psid"].Values).To(Equal([]string{"MT_000001"})) + again, err := DeviceToOutput(restored, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(again).To(Equal(w)) + } + }) + + ginkgo.It("does not promote an ID assigned before a failed identity recheck", func() { + d := outputDeviceFixture() + d.ObservationState, d.ReasonCode = "unavailable", "INVENTORY_CHANGED" + w, err := DeviceToOutput(d, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(w.ID).To(BeEmpty()) + Expect(w.SerialNumber).To(Equal(d.SerialNumber)) + Expect(w.PSID).To(BeEmpty()) + Expect(w.BMCAdministratorVerified).To(BeFalse()) + Expect(w.FirmwareInventoryComplete).To(BeFalse()) + Expect(w.Issues).To(ContainElement(DeviceIssue{Field: "identity", Code: "INVENTORY_CHANGED"})) + restored, err := DeviceFromOutput(w, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(restored.ID).To(BeEmpty()) + Expect(restored.ObservationState).NotTo(Equal("observed")) + Expect(restored.Observations["psid"].State).NotTo(Equal("observed")) + }) + + ginkgo.DescribeTable("preserves rejected PSID evidence independently of a matching SSH measurement", func(code string) { + d := outputDeviceFixture() + d.Observations["psid"] = Observation{State: "unknown", ReasonCode: code, Tool: "redfish"} + d.Runtime = &RuntimeObservation{Hardware: &HardwareObservation{Cards: []CardHardware{{Domain: "host-pci", PCIAddress: "0000:21:00.0", SerialNumber: d.SerialNumber, PSID: Observation{State: "observed", Values: []string{"MT_000001"}}}}}} + w, err := DeviceToOutput(d, "dpus") + Expect(err).NotTo(HaveOccurred()) + restored, err := DeviceFromOutput(w, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(restored.Observations["psid"].ReasonCode).To(Equal(code)) + Expect(restored.Observations["psid"].State).NotTo(Equal("observed")) + Expect(restored.Runtime.Hardware.Cards[0].PSID.Values).To(Equal([]string{"MT_000001"})) + }, ginkgo.Entry("conflict", "PSID_CONFLICT"), ginkgo.Entry("changed", "PSID_CHANGED"), ginkgo.Entry("invalid", "PSID_INVALID"), ginkgo.Entry("late recheck", "PSID_RECHECK_FAILED")) + + ginkgo.It("retains named next-boot values, explicit zero, host restrictions and real failures", func() { + d := outputDeviceFixture() + d.Observations["psid"] = Observation{State: "unknown", ReasonCode: "PSID_NOT_EXPOSED"} + d.Runtime = &RuntimeObservation{Observations: map[string]Observation{ + "board-product": {State: "observed", Values: []string{"B4240"}}, "mft-version": {State: "observed", Values: []string{"4.33.0-169"}}, + "hostname": {State: "observed"}, "neighbors": {State: "unavailable", ReasonCode: "TOOL_NOT_AVAILABLE", Tool: "ip"}, + }, Hardware: &HardwareObservation{Cards: []CardHardware{{Domain: "arm-pci", PCIAddress: "0002:21:00.0", Firmware: Observation{State: "observed", Values: []string{"32.4"}}, + NVConfig: map[string]Observation{ + "INTERNAL_CPU_OFFLOAD_ENGINE": {State: "observed", Values: []string{"next-boot=ENABLED(1)"}}, + "NUM_OF_VFS": {State: "observed", Values: []string{"next-boot=0"}}, + }, Privilege: Observation{State: "observed", Values: []string{"host[0].level=RESTRICTED", "host[0].disable_rshim=FALSE", "host[1].level=PRIVILEGED"}}, + }}}} + w, err := DeviceToOutput(d, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(w.Runtime.MFTVersion).To(Equal("4.33.0-169")) + Expect(*w.Runtime.Hardware.Cards[0].NextBoot.NumVFs).To(BeZero()) + Expect(*w.Runtime.Hardware.Cards[0].HostPrivileges[0].DisableRShim).To(BeFalse()) + Expect(w.Runtime.Hardware.Cards[0].HostPrivileges[1].DisableRShim).To(BeNil()) + Expect(w.Issues).To(ContainElement(DeviceIssue{Field: "runtime.neighbors", Code: "TOOL_NOT_AVAILABLE", Tool: "ip"})) + Expect(hasIssue(w.Issues, "runtime.hostname")).To(BeFalse()) + restored, err := DeviceFromOutput(w, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(restored.Runtime.Hardware.Cards[0].NVConfig["NUM_OF_VFS"].Values).To(Equal([]string{"next-boot=0"})) + Expect(restored.Runtime.Hardware.Cards[0].NVConfig["INTERNAL_CPU_RSHIM"].State).To(BeEmpty()) + Expect(restored.Runtime.Hardware.Cards[0].Privilege.Values).To(ContainElement("host[0].disable_rshim=FALSE")) + }) + + ginkgo.It("exports host rshim and rejects promoting failed adapter serial context", func() { + d := Device{ID: "host-one", RecordKey: "host-record", ObservationState: "observed", SSHEndpoint: "ssh://192.0.2.2:22", ReaderAdapter: "linux-host-ssh-v1", + Adapters: []Adapter{{Key: "card-one", SerialNumber: "candidate", IdentityState: "unknown", ReasonCode: "CARD_SERIAL_MISSING"}}, + Runtime: &RuntimeObservation{Hardware: &HardwareObservation{RShim: Observation{State: "observed", Values: []string{"LoadState=loaded", "ActiveState=active"}}}}} + w, err := DeviceToOutput(d, "hosts") + Expect(err).NotTo(HaveOccurred()) + Expect(w.Runtime.Hardware.RShim.ActiveState).To(Equal("active")) + restored, err := DeviceFromOutput(w, "hosts") + Expect(err).NotTo(HaveOccurred()) + Expect(restored.Adapters[0].SerialNumber).To(BeEmpty()) + Expect(restored.Adapters[0].SerialNumberRaw).To(Equal("candidate")) + Expect(restored.Adapters[0].IdentityState).To(Equal("unknown")) + Expect(restored.Adapters[0].ReasonCode).To(Equal("CARD_SERIAL_MISSING")) + }) + + ginkgo.It("keeps component and activation conflicts ineligible across the boundary", func() { + d := outputDeviceFixture() + d.Firmware = []Firmware{ + {Component: "nic", RawVersion: "32.1", State: "conflict", ReasonCode: "FIRMWARE_VERSION_CONFLICT", Activation: "running"}, + {Component: "nic", RawVersion: "32.2", State: "conflict", ReasonCode: "FIRMWARE_VERSION_CONFLICT", Activation: "unknown", ActivationReasonCode: "ACTIVATION_CONFLICT"}, + } + w, err := DeviceToOutput(d, "dpus") + Expect(err).NotTo(HaveOccurred()) + restored, err := DeviceFromOutput(w, "dpus") + Expect(err).NotTo(HaveOccurred()) + Expect(restored.Firmware).To(HaveLen(2)) + for _, f := range restored.Firmware { + Expect(f.State).To(Equal("conflict")) + } + Expect(restored.Firmware[0].ActivationReasonCode).To(Equal("ACTIVATION_CONFLICT")) + }) + + ginkgo.It("rejects inconsistent source, authorization, component and configuration claims", func() { + d := outputDeviceFixture() + d.ReaderAdapter = "unsupported-reader" + _, err := DeviceToOutput(d, "dpus") + Expect(err).To(HaveOccurred()) + w, err := DeviceToOutput(outputDeviceFixture(), "dpus") + Expect(err).NotTo(HaveOccurred()) + for _, mutate := range []func(*DeviceOutput){ + func(w *DeviceOutput) { + w.Issues = append(w.Issues, DeviceIssue{Field: "identity", Code: "INVENTORY_CHANGED"}) + }, + func(w *DeviceOutput) { w.SSHEndpoint = "ssh://192.0.2.3:22" }, + func(w *DeviceOutput) { w.PSID = "invalid" }, + func(w *DeviceOutput) { w.Issues = append(w.Issues, DeviceIssue{Field: "psid", Code: "PSID_CONFLICT"}) }, + func(w *DeviceOutput) { + w.Firmware = &FirmwareOutput{NIC: []FirmwareVersionOutput{{Version: "32.1", Activation: "running"}, {Version: "32.2", Activation: "running"}}} + }, + func(w *DeviceOutput) { + w.Firmware = &FirmwareOutput{NIC: []FirmwareVersionOutput{{Version: "32.1", Activation: "unknown"}}} + }, + } { + copy := w + mutate(©) + _, err := DeviceFromOutput(copy, "dpus") + Expect(err).To(HaveOccurred()) + } + }) +}) diff --git a/pkg/dpf/inventory/encoding.go b/pkg/dpf/inventory/encoding.go new file mode 100644 index 00000000..4ea5d7b1 --- /dev/null +++ b/pkg/dpf/inventory/encoding.go @@ -0,0 +1,346 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "bytes" + "encoding/json" + "fmt" + "maps" + "reflect" + "slices" + "strings" + + yaml "gopkg.in/yaml.v3" +) + +// inventoryWire avoids recursively invoking the Inventory codecs. Device itself +// keeps its ordinary encoding for consumers with their own versioned contracts. +type inventoryWire Inventory + +func (i Inventory) MarshalJSON() ([]byte, error) { + if i.FormatVersion == FormatVersion { + wire, err := i.output() + if err != nil { + return nil, err + } + return json.Marshal(wire) + } + wire, err := i.compact() + if err != nil { + return nil, err + } + return json.Marshal(wire) +} + +func (i Inventory) MarshalYAML() (any, error) { + var wire any + var err error + if i.FormatVersion == FormatVersion { + wire, err = i.output() + } else { + wire, err = i.compact() + } + if err != nil { + return nil, err + } + var node yaml.Node + if err := node.Encode(wire); err != nil { + return nil, err + } + // Evidence is internal diagnostic bookkeeping, not a deployment input. + // Strip it at the config boundary, including nested management, SSH, + // address-lookup and probe records. The source model and JSON diagnostic + // report remain untouched; legacy configs can still be read. + removeConfigEvidence(&node, reflect.TypeOf(wire)) + return &node, nil +} + +// removeConfigEvidence drops only typed provenance fields. Mapping keys in +// user data maps are always retained, even when they happen to be named +// evidence/evidenceRef/evidenceRefs. +func removeConfigEvidence(node *yaml.Node, typ reflect.Type) { + if node == nil || typ == nil { + return + } + for typ.Kind() == reflect.Pointer { + typ = typ.Elem() + } + switch node.Kind { + case yaml.MappingNode: + switch typ.Kind() { + case reflect.Map: + for index := 1; index < len(node.Content); index += 2 { + removeConfigEvidence(node.Content[index], typ.Elem()) + } + case reflect.Struct: + fields := configYAMLFields(typ) + kept := node.Content[:0] + for index := 0; index < len(node.Content); index += 2 { + key, value := node.Content[index], node.Content[index+1] + field, exists := fields[key.Value] + if exists && isConfigEvidenceField(field.name) { + continue + } + if exists { + removeConfigEvidence(value, field.typ) + } + kept = append(kept, key, value) + } + node.Content = kept + } + case yaml.SequenceNode: + if typ.Kind() == reflect.Slice || typ.Kind() == reflect.Array { + for _, child := range node.Content { + removeConfigEvidence(child, typ.Elem()) + } + } + } +} + +type configYAMLField struct { + name string + typ reflect.Type +} + +func configYAMLFields(typ reflect.Type) map[string]configYAMLField { + fields := make(map[string]configYAMLField, typ.NumField()) + for index := 0; index < typ.NumField(); index++ { + field := typ.Field(index) + tag := field.Tag.Get("yaml") + parts := strings.Split(tag, ",") + name := parts[0] + if name == "-" { + continue + } + inline := slices.Contains(parts[1:], "inline") + if inline { + inlineType := field.Type + for inlineType.Kind() == reflect.Pointer { + inlineType = inlineType.Elem() + } + if inlineType.Kind() == reflect.Struct { + for key, value := range configYAMLFields(inlineType) { + fields[key] = value + } + } + continue + } + if name == "" { + name = strings.ToLower(field.Name) + } + fields[name] = configYAMLField{name: name, typ: field.Type} + } + return fields +} + +func isConfigEvidenceField(name string) bool { + switch name { + case "evidence", "evidenceRef", "evidenceRefs": + return true + default: + return false + } +} + +func (i *Inventory) UnmarshalJSON(data []byte) error { + if i == nil { + return fmt.Errorf("nil inventory destination") + } + var header struct { + FormatVersion string `json:"formatVersion"` + } + if err := json.Unmarshal(data, &header); err != nil { + return err + } + decoder := json.NewDecoder(bytes.NewReader(data)) + decoder.DisallowUnknownFields() + if header.FormatVersion == FormatVersion { + var wire inventoryOutput + if err := decoder.Decode(&wire); err != nil { + return err + } + result, err := wire.inventory() + if err != nil { + return err + } + *i = result + return nil + } + var wire inventoryWire + if err := decoder.Decode(&wire); err != nil { + return err + } + result := Inventory(wire) + if err := NormalizeInventory(&result); err != nil { + return err + } + *i = result + return nil +} + +// UnmarshalYAML selects a versioned public schema and preserves strict nested +// validation. A custom node decoder must not bypass the caller's KnownFields. +func (i *Inventory) UnmarshalYAML(node *yaml.Node) error { + if i == nil { + return fmt.Errorf("nil inventory destination") + } + var header struct { + FormatVersion string `yaml:"formatVersion"` + } + if err := node.Decode(&header); err != nil { + return err + } + data, err := yaml.Marshal(node) + if err != nil { + return err + } + decoder := yaml.NewDecoder(bytes.NewReader(data)) + decoder.KnownFields(true) + if header.FormatVersion == FormatVersion { + var wire inventoryOutput + if err := decoder.Decode(&wire); err != nil { + return err + } + result, err := wire.inventory() + if err != nil { + return err + } + *i = result + return nil + } + var wire inventoryWire + if err := decoder.Decode(&wire); err != nil { + return err + } + result := Inventory(wire) + if err := NormalizeInventory(&result); err != nil { + return err + } + *i = result + return nil +} + +func (i Inventory) compact() (inventoryWire, error) { + wire := inventoryWire(i) + if i.FormatVersion != "v1alpha2" { + return wire, nil + } + var err error + wire.Hosts, err = compactDevices(i.Hosts) + if err != nil { + return inventoryWire{}, err + } + wire.DPUs, err = compactDevices(i.DPUs) + return wire, err +} + +// compactDevices projects diagnostics without changing the retained observations +// used by discovery checks. Typed facts retain successful evidence needed later. +func compactDevices(devices []Device) ([]Device, error) { + out := slices.Clone(devices) + for index := range out { + d := &out[index] + d.Observations = maps.Clone(d.Observations) + if err := restoreFacts(d); err != nil { + return nil, err + } + switch d.ObservationState { + case "observed": + d.ObservationState = "" + case "": + // Absence has a success meaning only in the compact format. An + // unknown in-memory record must therefore remain explicit. + d.ObservationState = "unknown" + } + for key, observation := range d.Observations { + if observation.State != "observed" { + if key == "client-certificate-support" && observation.State == "unknown" && observation.ReasonCode == "NOT_EVALUATED" { + delete(d.Observations, key) + } + continue + } + switch key { + case "psid": + if len(observation.Values) != 1 || observation.Values[0] == "" || observation.ReasonCode != "PSID_OBSERVED" || observation.Tool != "redfish" { + return nil, fmt.Errorf("device %s has invalid verified PSID evidence", d.RecordKey) + } + d.PSID = observation.Values[0] + case "bmc-administrator": + if observation.ReasonCode == "BMC_ADMIN_VERIFIED" { + d.Authentication.BMCAdministrator = "verified" + } + case "firmware-inventory": + d.FirmwareInventoryComplete = true + } + delete(d.Observations, key) + } + } + return out, nil +} + +// NormalizeInventory restores the internal observation model from the compact +// v1alpha2 wire contract. Both codecs call it after strict decoding. +// Legacy records retain their exact state, including absent/unknown evidence. +func NormalizeInventory(i *Inventory) error { + if i == nil || i.FormatVersion != "v1alpha2" { + return nil + } + for _, devices := range [][]Device{i.Hosts, i.DPUs} { + for index := range devices { + d := &devices[index] + if d.ObservationState == "" { + d.ObservationState = "unknown" + if d.ID != "" { + d.ObservationState = "observed" + } + } + if err := restoreFacts(d); err != nil { + return err + } + } + } + return nil +} + +// restoreFacts is shared by reading and writing so contradictory wire facts are +// rejected even when a caller constructs an Inventory directly for encoding. +func restoreFacts(d *Device) error { + if d.Authentication.BMCAdministrator != "" { + if d.Authentication.BMCAdministrator != "verified" { + return fmt.Errorf("device %s has invalid BMC administrator evidence", d.RecordKey) + } + if err := restoreObservation(d, "bmc-administrator", Observation{State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED"}); err != nil { + return err + } + d.Authentication.BMCAdministrator = "" + } + if d.PSID != "" { + if err := restoreObservation(d, "psid", Observation{State: "observed", Values: []string{d.PSID}, ReasonCode: "PSID_OBSERVED", Tool: "redfish"}); err != nil { + return err + } + d.PSID = "" + } + if d.FirmwareInventoryComplete { + if err := restoreObservation(d, "firmware-inventory", Observation{State: "observed"}); err != nil { + return err + } + d.FirmwareInventoryComplete = false + } + return nil +} + +func restoreObservation(d *Device, key string, fact Observation) error { + if old, exists := d.Observations[key]; exists { + if old.State != fact.State || old.ReasonCode != fact.ReasonCode || !slices.Equal(old.Values, fact.Values) || (key == "psid" && old.Tool != fact.Tool) { + return fmt.Errorf("device %s has conflicting %s evidence", d.RecordKey, key) + } + return nil + } + if d.Observations == nil { + d.Observations = map[string]Observation{} + } + d.Observations[key] = fact + return nil +} diff --git a/pkg/dpf/inventory/encoding_test.go b/pkg/dpf/inventory/encoding_test.go new file mode 100644 index 00000000..55f029ca --- /dev/null +++ b/pkg/dpf/inventory/encoding_test.go @@ -0,0 +1,148 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "encoding/json" + "testing" + + "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + yaml "gopkg.in/yaml.v3" +) + +func TestInventory(t *testing.T) { + RegisterFailHandler(ginkgo.Fail) + ginkgo.RunSpecs(t, "DPF inventory encoding") +} + +func compactFixture() Inventory { + return Inventory{FormatVersion: "v1alpha2", Hosts: []Device{{ID: "host-one", ObservationState: "observed"}}, DPUs: []Device{{ + ID: "dpu-one", RecordKey: "record-one", ObservationState: "observed", BMCEndpoint: "https://192.0.2.1", + Observations: map[string]Observation{ + "psid": {State: "observed", Values: []string{"MT_0000001774"}, ReasonCode: "PSID_OBSERVED", Tool: "redfish"}, + "bmc-administrator": {State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED", Tool: "redfish"}, + "firmware-inventory": {State: "observed"}, + "managers": {State: "observed"}, + "client-certificate-support": {State: "unknown", ReasonCode: "NOT_EVALUATED"}, + "bmc-time": {State: "unknown", ReasonCode: "PROPERTY_NOT_EXPOSED", Message: "DateTime is absent"}, + "bmc-ntp": {State: "unavailable", ReasonCode: "HTTP_FORBIDDEN", Message: "Read denied"}, + }, + }}} +} + +var _ = ginkgo.Describe("compact inventory", func() { + ginkgo.It("writes only diagnostics in device observations without mutating retained evidence", func() { + in := compactFixture() + before, err := json.Marshal(inventoryWire(in)) + Expect(err).NotTo(HaveOccurred()) + encoded, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + var wire inventoryWire + Expect(json.Unmarshal(encoded, &wire)).To(Succeed()) + Expect(wire.DPUs[0].Observations).To(HaveLen(2)) + Expect(wire.DPUs[0].Observations).To(HaveKey("bmc-time")) + Expect(wire.DPUs[0].Observations).To(HaveKey("bmc-ntp")) + Expect(wire.DPUs[0].PSID).To(Equal("MT_0000001774")) + Expect(wire.DPUs[0].Authentication.BMCAdministrator).To(Equal("verified")) + Expect(wire.DPUs[0].FirmwareInventoryComplete).To(BeTrue()) + Expect(string(encoded)).NotTo(ContainSubstring("observationState")) + after, err := json.Marshal(inventoryWire(in)) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(before)) + }) + + ginkgo.It("restores identity and required positive evidence from JSON and YAML", func() { + in := compactFixture() + encoded, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + var fromJSON Inventory + Expect(json.Unmarshal(encoded, &fromJSON)).To(Succeed()) + encodedYAML, err := yaml.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + var fromYAML Inventory + Expect(yaml.Unmarshal(encodedYAML, &fromYAML)).To(Succeed()) + Expect(NormalizeInventory(&fromYAML)).To(Succeed()) + for _, restored := range []Inventory{fromJSON, fromYAML} { + Expect(restored.DPUs[0].ObservationState).To(Equal("observed")) + Expect(restored.Hosts[0].ObservationState).To(Equal("observed")) + Expect(restored.DPUs[0].Observations["psid"]).To(Equal(in.DPUs[0].Observations["psid"])) + Expect(restored.DPUs[0].Observations["bmc-administrator"].ReasonCode).To(Equal("BMC_ADMIN_VERIFIED")) + Expect(restored.DPUs[0].Observations["firmware-inventory"].State).To(Equal("observed")) + Expect(NormalizeInventory(&restored)).To(Succeed()) + } + again, err := json.Marshal(fromJSON) + Expect(err).NotTo(HaveOccurred()) + Expect(again).To(Equal(encoded)) + againYAML, err := yaml.Marshal(fromYAML) + Expect(err).NotTo(HaveOccurred()) + Expect(againYAML).To(Equal(encodedYAML)) + }) + + ginkgo.It("preserves explicit failures and unknown records even when an ID was assigned", func() { + in := Inventory{FormatVersion: "v1alpha2", DPUs: []Device{ + {ID: "before-recheck", ObservationState: "unavailable", ReasonCode: "IDENTITY_CHANGED"}, + {ID: "not-verified", ObservationState: ""}, + {ObservationState: "observed"}, + }} + encoded, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + var restored Inventory + Expect(json.Unmarshal(encoded, &restored)).To(Succeed()) + Expect(restored.DPUs[0].ObservationState).To(Equal("unavailable")) + Expect(restored.DPUs[0].ReasonCode).To(Equal("IDENTITY_CHANGED")) + Expect(restored.DPUs[1].ObservationState).To(Equal("unknown")) + Expect(restored.DPUs[2].ObservationState).To(Equal("unknown")) + }) + + ginkgo.It("retains legacy encoding and never infers missing legacy state or proof", func() { + in := compactFixture() + in.FormatVersion = "v1alpha1" + in.Hosts[0].ObservationState = "" + encoded, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + Expect(string(encoded)).To(ContainSubstring(`"observationState":"observed"`)) + var restored Inventory + Expect(json.Unmarshal(encoded, &restored)).To(Succeed()) + Expect(restored.Hosts[0].ObservationState).To(BeEmpty()) + Expect(restored.DPUs[0].Observations).To(Equal(in.DPUs[0].Observations)) + }) + + ginkgo.DescribeTable("rejects conflicting compact positive facts", func(extra string) { + var result Inventory + data := `{"formatVersion":"v1alpha2","dpus":[{"id":"dpu-one",` + extra + `}]}` + Expect(json.Unmarshal([]byte(data), &result)).To(MatchError(ContainSubstring("conflicting"))) + var raw inventoryWire + Expect(json.Unmarshal([]byte(data), &raw)).To(Succeed()) + before, err := json.Marshal(raw) + Expect(err).NotTo(HaveOccurred()) + _, err = json.Marshal(Inventory(raw)) + Expect(err).To(MatchError(ContainSubstring("conflicting"))) + after, err := json.Marshal(raw) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(before)) + }, + ginkgo.Entry("administrator", `"authentication":{"bmcAdministrator":"verified"},"observations":{"bmc-administrator":{"state":"failed","reasonCode":"BMC_ADMIN_REQUIRED"}}`), + ginkgo.Entry("PSID", `"psid":"MT_A","observations":{"psid":{"state":"unavailable","reasonCode":"PSID_CHANGED"}}`), + ginkgo.Entry("different successful PSID", `"psid":"MT_A","observations":{"psid":{"state":"observed","values":["MT_B"],"reasonCode":"PSID_OBSERVED","tool":"redfish"}}`), + ginkgo.Entry("firmware source", `"firmwareInventoryComplete":true,"observations":{"firmware-inventory":{"state":"unavailable","reasonCode":"HTTP_FORBIDDEN"}}`), + ) + + ginkgo.It("retains failures whose reason happens to say not evaluated", func() { + in := Inventory{FormatVersion: "v1alpha2", Hosts: []Device{{ID: "host-one", ObservationState: "observed", Observations: map[string]Observation{ + "client-certificate-support": {State: "failed", ReasonCode: "NOT_EVALUATED"}, + "other-fact": {State: "unknown", ReasonCode: "NOT_EVALUATED"}, + }}}} + encoded, err := json.Marshal(in) + Expect(err).NotTo(HaveOccurred()) + var restored Inventory + Expect(json.Unmarshal(encoded, &restored)).To(Succeed()) + Expect(restored.Hosts[0].Observations).To(Equal(in.Hosts[0].Observations)) + }) + + ginkgo.It("keeps JSON unknown-field validation inside nested inventory records", func() { + var result Inventory + Expect(json.Unmarshal([]byte(`{"formatVersion":"v1alpha2","dpus":[{"authentication":{"administratorTypo":"verified"}}]}`), &result)).To(HaveOccurred()) + }) +}) diff --git a/pkg/dpf/inventory/hardware.go b/pkg/dpf/inventory/hardware.go new file mode 100644 index 00000000..1ed8baa4 --- /dev/null +++ b/pkg/dpf/inventory/hardware.go @@ -0,0 +1,62 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package inventory + +// HardwareObservation keeps PCI-domain-local facts separate from device identity. +// A host or ARM may expose multiple independent cards; PCI addresses are not +// comparable across those domains. +type HardwareObservation struct { + Cards []CardHardware `json:"cards" yaml:"cards"` + RShim Observation `json:"rshim" yaml:"rshim"` +} + +type CardHardware struct { + PCIAddress string `json:"pciAddress" yaml:"pciAddress"` + Domain string `json:"domain" yaml:"domain"` + Model string `json:"model" yaml:"model"` + SerialNumber string `json:"serialNumber,omitempty" yaml:"serialNumber,omitempty"` + Firmware Observation `json:"firmware" yaml:"firmware"` + PSID Observation `json:"psid" yaml:"psid"` + NVConfig map[string]Observation `json:"nvConfig" yaml:"nvConfig"` + Privilege Observation `json:"privilege" yaml:"privilege"` + EvidenceRefs []string `json:"evidenceRefs" yaml:"evidenceRefs"` +} + +type NetworkObservation struct { + Links []NetworkLink `json:"links" yaml:"links"` + Resolver ResolverObservation `json:"resolver" yaml:"resolver"` + Neighbors []Neighbor `json:"neighbors" yaml:"neighbors"` + LLDP []LLDPNeighbor `json:"lldp" yaml:"lldp"` +} + +type NetworkLink struct { + Name string `json:"name" yaml:"name"` + MAC string `json:"mac,omitempty" yaml:"mac,omitempty"` + Master string `json:"master,omitempty" yaml:"master,omitempty"` + Kind string `json:"kind,omitempty" yaml:"kind,omitempty"` +} + +type ResolverObservation struct { + Owner string `json:"owner" yaml:"owner"` + Path string `json:"path,omitempty" yaml:"path,omitempty"` + Search []string `json:"search" yaml:"search"` + PerLink map[string][]string `json:"perLink" yaml:"perLink"` + Domains map[string][]string `json:"domains" yaml:"domains"` + DNSMasqNoResolv *bool `json:"dnsmasqNoResolv,omitempty" yaml:"dnsmasqNoResolv,omitempty"` + DNSMasqServers []string `json:"dnsmasqServers" yaml:"dnsmasqServers"` +} + +// Neighbor is an observed address candidate, never device identity or authority +// to connect. Its state and collection vantage must be retained by consumers. +type Neighbor struct { + Address string `json:"address" yaml:"address"` + MAC string `json:"mac,omitempty" yaml:"mac,omitempty"` + Interface string `json:"interface" yaml:"interface"` + State []string `json:"state" yaml:"state"` +} + +type LLDPNeighbor struct { + Interface string `json:"interface" yaml:"interface"` + Chassis string `json:"chassis,omitempty" yaml:"chassis,omitempty"` + Port string `json:"port,omitempty" yaml:"port,omitempty"` +} diff --git a/pkg/dpf/inventory/identity.go b/pkg/dpf/inventory/identity.go new file mode 100644 index 00000000..2348f9e3 --- /dev/null +++ b/pkg/dpf/inventory/identity.go @@ -0,0 +1,54 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package inventory + +// LegacyHostCardBindings identifies preview handoffs whose Redfish card record +// key changed with the serial. Those records cannot safely guard a refresh and +// require a fresh seed and output; do not silently migrate physical bindings. +func LegacyHostCardBindings(old *Inventory) bool { + if old == nil { + return false + } + derived := map[string]bool{} + for _, d := range old.DPUs { + if d.ReaderAdapter == "host-card-v1" { + derived[d.RecordKey] = true + } + } + for _, host := range old.Hosts { + for _, card := range host.Adapters { + if card.SerialNumber != "" && card.Key == Digest("dpf-adapter-v1", []string{host.ID, card.SerialNumber}) && + derived[Digest("dpf-host-card-v1", []string{host.ID, card.Key})] { + return true + } + } + } + return false +} + +// IdentityConflict rejects replacement of a previously observed physical binding +// and legacy bindings that cannot support a safe comparison. +func IdentityConflict(old, current *Inventory) bool { + if old == nil || current == nil { + return false + } + if LegacyHostCardBindings(old) { + return true + } + before := map[string]string{} + for _, group := range [][]Device{old.DPUs, old.Hosts} { + for _, d := range group { + if d.ID != "" { + before[d.RecordKey] = d.ID + } + } + } + for _, group := range [][]Device{current.DPUs, current.Hosts} { + for _, d := range group { + if id := before[d.RecordKey]; id != "" && d.ID != "" && id != d.ID { + return true + } + } + } + return false +} diff --git a/pkg/dpf/inventory/output.go b/pkg/dpf/inventory/output.go new file mode 100644 index 00000000..fdc03b1b --- /dev/null +++ b/pkg/dpf/inventory/output.go @@ -0,0 +1,111 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package inventory + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" +) + +// inventoryOutput is the public v3 handoff. Collection bookkeeping stays in the +// collector model; required facts and failures have explicit domain fields here. +type inventoryOutput struct { + Software *releases.Summary `json:"software,omitempty" yaml:"software,omitempty"` + AddressLookup *AddressLookup `json:"addressLookup,omitempty" yaml:"addressLookup,omitempty"` + FormatVersion string `json:"formatVersion" yaml:"formatVersion"` + Collection Collection `json:"collection" yaml:"collection"` + ScopeDigest string `json:"scopeDigest,omitempty" yaml:"scopeDigest,omitempty"` + Fingerprint string `json:"fingerprint,omitempty" yaml:"fingerprint,omitempty"` + Management *Management `json:"management,omitempty" yaml:"management,omitempty"` + DPUs []DeviceOutput `json:"dpus,omitempty" yaml:"dpus,omitempty"` + Hosts []DeviceOutput `json:"hosts,omitempty" yaml:"hosts,omitempty"` + Associations []associationOutput `json:"associations,omitempty" yaml:"associations,omitempty"` + Probes []ProbeResult `json:"probes,omitempty" yaml:"probes,omitempty"` + Evidence map[string]Evidence `json:"evidence,omitempty" yaml:"evidence,omitempty"` +} + +type associationOutput struct { + DPUID string `json:"dpuID" yaml:"dpuID"` + HostID string `json:"hostID" yaml:"hostID"` + HostAdapterKey string `json:"hostAdapterKey" yaml:"hostAdapterKey"` + SerialNumber string `json:"serialNumber" yaml:"serialNumber"` +} + +func (i Inventory) output() (inventoryOutput, error) { + out := inventoryOutput{ + FormatVersion: FormatVersion, Collection: i.Collection, + ScopeDigest: i.ScopeDigest, Fingerprint: i.Fingerprint, + Management: i.Management, AddressLookup: i.AddressLookup, Probes: i.Probes, + } + if i.Software != nil { + summary := releases.Summarize(*i.Software) + out.Software = &summary + } + for _, group := range []struct { + role string + devices []Device + output *[]DeviceOutput + }{{"dpus", i.DPUs, &out.DPUs}, {"hosts", i.Hosts, &out.Hosts}} { + for _, device := range group.devices { + value, err := DeviceToOutput(device, group.role) + if err != nil { + return inventoryOutput{}, err + } + *group.output = append(*group.output, value) + } + } + for _, association := range i.Associations { + out.Associations = append(out.Associations, associationOutput{ + DPUID: association.DPUID, HostID: association.HostID, + HostAdapterKey: association.HostAdapterKey, SerialNumber: association.SerialNumber, + }) + } + // Device source diagnostics are carried with their fields. Management/probe + // evidence has its own contract and is independent of the host/DPU records. + for key, evidence := range i.Evidence { + if evidence.Kind == "redfish" { + continue + } + if out.Evidence == nil { + out.Evidence = map[string]Evidence{} + } + out.Evidence[key] = evidence + } + return out, nil +} + +func (w inventoryOutput) inventory() (Inventory, error) { + out := Inventory{ + FormatVersion: FormatVersion, Collection: w.Collection, + ScopeDigest: w.ScopeDigest, Fingerprint: w.Fingerprint, + Management: w.Management, AddressLookup: w.AddressLookup, + Probes: w.Probes, Evidence: w.Evidence, + } + if w.Software != nil { + release, err := w.Software.Release() + if err != nil { + return Inventory{}, err + } + out.Software = &release + } + for _, group := range []struct { + role string + devices []DeviceOutput + output *[]Device + }{{"dpus", w.DPUs, &out.DPUs}, {"hosts", w.Hosts, &out.Hosts}} { + for _, device := range group.devices { + value, err := DeviceFromOutput(device, group.role) + if err != nil { + return Inventory{}, err + } + *group.output = append(*group.output, value) + } + } + for _, association := range w.Associations { + out.Associations = append(out.Associations, Association{ + DPUID: association.DPUID, HostID: association.HostID, + HostAdapterKey: association.HostAdapterKey, SerialNumber: association.SerialNumber, + }) + } + return out, nil +} diff --git a/pkg/dpf/inventory/probe.go b/pkg/dpf/inventory/probe.go new file mode 100644 index 00000000..88364cc4 --- /dev/null +++ b/pkg/dpf/inventory/probe.go @@ -0,0 +1,31 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package inventory + +type Interface struct { + Name string `json:"name" yaml:"name"` + Index int `json:"index" yaml:"index"` + MTU int `json:"mtu" yaml:"mtu"` + Addresses []string `json:"addresses" yaml:"addresses"` +} +type Route struct { + Destination string `json:"destination" yaml:"destination"` + Gateway string `json:"gateway,omitempty" yaml:"gateway,omitempty"` + InterfaceIndex int `json:"interfaceIndex" yaml:"interfaceIndex"` + Table int `json:"table" yaml:"table"` +} +type ProbeAddress struct { + Address string `json:"address" yaml:"address"` + TCP string `json:"tcp" yaml:"tcp"` + TLS string `json:"tls" yaml:"tls"` + HTTP string `json:"http" yaml:"http"` + HTTPStatus int `json:"httpStatus,omitempty" yaml:"httpStatus,omitempty"` +} +type ProbeResult struct { + NodeUID string `json:"nodeUID" yaml:"nodeUID"` + NetworkMode string `json:"networkMode" yaml:"networkMode"` + Destination string `json:"destination" yaml:"destination"` + DNS string `json:"dns" yaml:"dns"` + Addresses []ProbeAddress `json:"addresses" yaml:"addresses"` + EvidenceRef string `json:"evidenceRef" yaml:"evidenceRef"` +} diff --git a/pkg/dpf/inventory/types.go b/pkg/dpf/inventory/types.go new file mode 100644 index 00000000..1d159bc6 --- /dev/null +++ b/pkg/dpf/inventory/types.go @@ -0,0 +1,259 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package inventory defines secret-free DPF observations and check results. +package inventory + +import ( + "crypto/sha256" + "encoding/json" + "fmt" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/releases" + "time" +) + +const FormatVersion = "v1alpha3" + +type Collection struct { + RunID string `json:"runID" yaml:"runID"` + StartedAt time.Time `json:"startedAt" yaml:"startedAt"` + FinishedAt time.Time `json:"finishedAt" yaml:"finishedAt"` + Mode string `json:"mode" yaml:"mode"` + Collector string `json:"collector" yaml:"collector"` +} +type ScopeEndpoint struct { + TLSTrustDigest string `json:"tlsTrustDigest,omitempty" yaml:"tlsTrustDigest,omitempty"` + SSHTrustDigest string `json:"sshTrustDigest,omitempty" yaml:"sshTrustDigest,omitempty"` + SSHSudo bool `json:"sshSudo,omitempty" yaml:"sshSudo,omitempty"` + Role string `json:"role" yaml:"role"` + Endpoint string `json:"endpoint" yaml:"endpoint"` + SystemURI string `json:"systemURI,omitempty" yaml:"systemURI,omitempty"` + TLSInsecureSkipVerify bool `json:"tlsInsecureSkipVerify,omitempty" yaml:"tlsInsecureSkipVerify,omitempty"` +} +type Evidence struct { + Namespace string `json:"namespace,omitempty" yaml:"namespace,omitempty"` + PodUID string `json:"podUID,omitempty" yaml:"podUID,omitempty"` + NodeUID string `json:"nodeUID,omitempty" yaml:"nodeUID,omitempty"` + ImageDigest string `json:"imageDigest,omitempty" yaml:"imageDigest,omitempty"` + DNSPolicy string `json:"dnsPolicy,omitempty" yaml:"dnsPolicy,omitempty"` + DestinationEndpoint string `json:"destinationEndpoint,omitempty" yaml:"destinationEndpoint,omitempty"` + Kind string `json:"kind" yaml:"kind"` + InputRef string `json:"inputRef,omitempty" yaml:"inputRef,omitempty"` + ResourceURI string `json:"resourceURI,omitempty" yaml:"resourceURI,omitempty"` + Property string `json:"property,omitempty" yaml:"property,omitempty"` + ObservedAt time.Time `json:"observedAt" yaml:"observedAt"` + Vantage string `json:"vantage" yaml:"vantage"` + Adapter string `json:"adapter,omitempty" yaml:"adapter,omitempty"` + Authenticated bool `json:"authenticated" yaml:"authenticated"` +} +type Firmware struct { + RawID string `json:"rawID,omitempty" yaml:"rawID,omitempty"` + RawName string `json:"rawName,omitempty" yaml:"rawName,omitempty"` + ActivationReasonCode string `json:"activationReasonCode,omitempty" yaml:"activationReasonCode,omitempty"` + ActivationMessage string `json:"activationMessage,omitempty" yaml:"activationMessage,omitempty"` + Message string `json:"message,omitempty" yaml:"message,omitempty"` + ResourceURI string `json:"resourceURI,omitempty" yaml:"resourceURI,omitempty"` + Component string `json:"component" yaml:"component"` + State string `json:"state" yaml:"state"` + RawVersion string `json:"rawVersion,omitempty" yaml:"rawVersion,omitempty"` + Activation string `json:"activation" yaml:"activation"` + ReasonCode string `json:"reasonCode,omitempty" yaml:"reasonCode,omitempty"` + EvidenceRefs []string `json:"evidenceRefs,omitempty" yaml:"evidenceRefs,omitempty"` +} +type Authentication struct { + BMCAdministrator string `json:"bmcAdministrator,omitempty" yaml:"bmcAdministrator,omitempty"` + PasswordRead string `json:"passwordRead" yaml:"passwordRead"` + Method string `json:"method,omitempty" yaml:"method,omitempty"` + ClientCertificateSupport string `json:"clientCertificateSupport" yaml:"clientCertificateSupport"` + DPFClientAuthentication string `json:"dpfClientAuthentication" yaml:"dpfClientAuthentication"` +} +type TimeObservation struct { + Message string `json:"message,omitempty" yaml:"message,omitempty"` + State string `json:"state" yaml:"state"` + DateTime string `json:"dateTime,omitempty" yaml:"dateTime,omitempty"` + Offset string `json:"offset,omitempty" yaml:"offset,omitempty"` + NTPEnabled *bool `json:"ntpEnabled,omitempty" yaml:"ntpEnabled,omitempty"` + NTPServers []string `json:"ntpServers,omitempty" yaml:"ntpServers,omitempty"` + ReasonCode string `json:"reasonCode,omitempty" yaml:"reasonCode,omitempty"` +} +type PCIFunction struct { + Address string `json:"address" yaml:"address"` + Driver string `json:"driver,omitempty" yaml:"driver,omitempty"` + NUMANode string `json:"numaNode,omitempty" yaml:"numaNode,omitempty"` + Netdevs []string `json:"netdevs,omitempty" yaml:"netdevs,omitempty"` +} +type Adapter struct { + PCIFunctions []PCIFunction `json:"pciFunctions,omitempty" yaml:"pciFunctions,omitempty"` + PCIAddress string `json:"pciAddress,omitempty" yaml:"pciAddress,omitempty"` + VendorID string `json:"vendorID,omitempty" yaml:"vendorID,omitempty"` + DeviceID string `json:"deviceID,omitempty" yaml:"deviceID,omitempty"` + IdentityState string `json:"identityState,omitempty" yaml:"identityState,omitempty"` + ReasonCode string `json:"reasonCode,omitempty" yaml:"reasonCode,omitempty"` + Key string `json:"adapterKey" yaml:"adapterKey"` + SerialNumber string `json:"serialNumber" yaml:"serialNumber"` + SerialNumberRaw string `json:"serialNumberRaw" yaml:"serialNumberRaw"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + Slot string `json:"slot,omitempty" yaml:"slot,omitempty"` + Selection string `json:"selection" yaml:"selection"` + EvidenceRefs []string `json:"evidenceRefs,omitempty" yaml:"evidenceRefs,omitempty"` +} + +// RuntimeObservation contains OS-visible facts, separate from BMC management readiness. +type RuntimeObservation struct { + Hardware *HardwareObservation `json:"hardware,omitempty" yaml:"hardware,omitempty"` + Network *NetworkObservation `json:"network,omitempty" yaml:"network,omitempty"` + Hostname string `json:"hostname,omitempty" yaml:"hostname,omitempty"` + OS string `json:"os,omitempty" yaml:"os,omitempty"` + Interfaces []Interface `json:"interfaces,omitempty" yaml:"interfaces,omitempty"` + Routes []Route `json:"routes,omitempty" yaml:"routes,omitempty"` + DNS []string `json:"dns,omitempty" yaml:"dns,omitempty"` + Observations map[string]Observation `json:"observations,omitempty" yaml:"observations,omitempty"` +} + +// Device represents every requested endpoint, including failed observations. +type Device struct { + PSID string `json:"psid,omitempty" yaml:"psid,omitempty"` + FirmwareInventoryComplete bool `json:"firmwareInventoryComplete,omitempty" yaml:"firmwareInventoryComplete,omitempty"` + Message string `json:"message,omitempty" yaml:"message,omitempty"` + Observations map[string]Observation `json:"observations,omitempty" yaml:"observations,omitempty"` + SSHEndpoint string `json:"sshEndpoint,omitempty" yaml:"sshEndpoint,omitempty"` + BMCAddressState string `json:"bmcAddressState,omitempty" yaml:"bmcAddressState,omitempty"` + Runtime *RuntimeObservation `json:"runtime,omitempty" yaml:"runtime,omitempty"` + RecordKey string `json:"recordKey" yaml:"recordKey"` + InputRef string `json:"inputRef" yaml:"inputRef"` + BMCEndpoint string `json:"bmcEndpoint" yaml:"bmcEndpoint"` + SystemURI string `json:"systemURI,omitempty" yaml:"systemURI,omitempty"` + ObservationState string `json:"observationState,omitempty" yaml:"observationState,omitempty"` + ReasonCode string `json:"reasonCode,omitempty" yaml:"reasonCode,omitempty"` + ID string `json:"id,omitempty" yaml:"id,omitempty"` + IdentityKey []string `json:"identityKey,omitempty" yaml:"identityKey,omitempty"` + SerialNumber string `json:"serialNumber,omitempty" yaml:"serialNumber,omitempty"` + SerialNumberRaw string `json:"serialNumberRaw,omitempty" yaml:"serialNumberRaw,omitempty"` + SystemUUID string `json:"systemUUID,omitempty" yaml:"systemUUID,omitempty"` + Manufacturer string `json:"manufacturer,omitempty" yaml:"manufacturer,omitempty"` + Model string `json:"model,omitempty" yaml:"model,omitempty"` + ReaderAdapter string `json:"readerAdapter,omitempty" yaml:"readerAdapter,omitempty"` + InventoryCompleteness string `json:"inventoryCompleteness,omitempty" yaml:"inventoryCompleteness,omitempty"` + Qualification string `json:"qualification" yaml:"qualification"` + Firmware []Firmware `json:"firmware,omitempty" yaml:"firmware,omitempty"` + Authentication Authentication `json:"authentication" yaml:"authentication"` + Time TimeObservation `json:"time,omitzero" yaml:"time,omitempty"` + Adapters []Adapter `json:"adapters,omitempty" yaml:"adapters,omitempty"` + AdvertisedResetTypes []string `json:"advertisedResetTypes,omitempty" yaml:"advertisedResetTypes,omitempty"` + EvidenceRefs []string `json:"evidenceRefs,omitempty" yaml:"evidenceRefs,omitempty"` +} +type Node struct { + Interfaces []Interface `json:"interfaces,omitempty" yaml:"interfaces,omitempty"` + Routes []Route `json:"routes,omitempty" yaml:"routes,omitempty"` + Name string `json:"name" yaml:"name"` + UID string `json:"uid" yaml:"uid"` + ControlPlane bool `json:"controlPlane" yaml:"controlPlane"` + Ready bool `json:"ready" yaml:"ready"` + Unschedulable bool `json:"unschedulable" yaml:"unschedulable"` + Architecture string `json:"architecture" yaml:"architecture"` + OS string `json:"os" yaml:"os"` + Addresses []string `json:"addresses" yaml:"addresses"` +} +type Observation struct { + Message string `json:"message,omitempty" yaml:"message,omitempty"` + Tool string `json:"tool,omitempty" yaml:"tool,omitempty"` + State string `json:"state" yaml:"state"` + Values []string `json:"values,omitempty" yaml:"values,omitempty"` + ReasonCode string `json:"reasonCode,omitempty" yaml:"reasonCode,omitempty"` +} +type Management struct { + Observations map[string]Observation `json:"observations,omitempty" yaml:"observations,omitempty"` + APIEndpoint string `json:"apiEndpoint" yaml:"apiEndpoint"` + SystemNamespaceUID string `json:"systemNamespaceUID" yaml:"systemNamespaceUID"` + TrustDigest string `json:"trustDigest" yaml:"trustDigest"` + KubernetesVersion string `json:"kubernetesVersion" yaml:"kubernetesVersion"` + Namespace string `json:"namespace" yaml:"namespace"` + Nodes []Node `json:"nodes" yaml:"nodes"` + EvidenceRefs []string `json:"evidenceRefs,omitempty" yaml:"evidenceRefs,omitempty"` +} +type Association struct { + DPUID string `json:"dpuID" yaml:"dpuID"` + HostID string `json:"hostID" yaml:"hostID"` + HostAdapterKey string `json:"hostAdapterKey" yaml:"hostAdapterKey"` + SerialNumber string `json:"serialNumber" yaml:"serialNumber"` + EvidenceRefs []string `json:"evidenceRefs,omitempty" yaml:"evidenceRefs,omitempty"` +} +type Inventory struct { + Software *releases.Release `json:"software,omitempty" yaml:"software,omitempty"` + AddressLookup *AddressLookup `json:"addressLookup,omitempty" yaml:"addressLookup,omitempty"` + FormatVersion string `json:"formatVersion" yaml:"formatVersion"` + Collection Collection `json:"collection" yaml:"collection"` + RequestedScope []ScopeEndpoint `json:"requestedScope" yaml:"requestedScope"` + ScopeDigest string `json:"scopeDigest,omitempty" yaml:"scopeDigest,omitempty"` + Fingerprint string `json:"fingerprint,omitempty" yaml:"fingerprint,omitempty"` + Management *Management `json:"management,omitempty" yaml:"management,omitempty"` + DPUs []Device `json:"dpus" yaml:"dpus"` + Hosts []Device `json:"hosts" yaml:"hosts"` + Associations []Association `json:"associations" yaml:"associations"` + Probes []ProbeResult `json:"probes,omitempty" yaml:"probes,omitempty"` + Evidence map[string]Evidence `json:"evidence,omitempty" yaml:"evidence,omitempty"` +} +type Check struct { + Message string `json:"message,omitempty"` + ID string `json:"id"` + Subject string `json:"subject"` + Vantage string `json:"vantage"` + RequiredPhase string `json:"requiredPhase"` + State string `json:"state"` + ReasonCode string `json:"reasonCode"` + EvidenceRefs []string `json:"evidenceRefs,omitempty"` +} +type Problem struct { + Tool string `json:"tool,omitempty"` + Method string `json:"method,omitempty"` + ResourceURI string `json:"resourceURI,omitempty"` + HTTPStatus int `json:"httpStatus,omitempty"` + Stage string `json:"stage,omitempty"` + Code string `json:"code"` + Message string `json:"message"` + InputRef string `json:"inputRef,omitempty"` + ExitCode int `json:"-"` +} +type Artifact struct { + Path string `json:"path"` + Kind string `json:"kind"` +} +type Change struct { + Path string `json:"path"` + State string `json:"state"` + Previous any `json:"previous,omitempty"` + Current any `json:"current,omitempty"` + RequiredPhase string `json:"requiredPhase"` +} +type Cleanup struct { + State string `json:"state"` + Leftovers []string `json:"leftovers"` +} +type Discovery struct { + Outcome string `json:"outcome"` + Mode string `json:"mode"` + Checks []Check `json:"checks"` + Cleanup Cleanup `json:"cleanup"` +} +type Report struct { + SchemaVersion string `json:"schemaVersion"` + Target string `json:"target"` + Phase string `json:"phase"` + Execution string `json:"execution"` + Acceptance string `json:"acceptance"` + Inventory *Inventory `json:"inventory,omitempty"` + Discovery Discovery `json:"discovery"` + Diff []Change `json:"diff"` + PendingInputs []Problem `json:"pendingInputs"` + Artifacts []Artifact `json:"artifacts"` + Errors []Problem `json:"errors"` +} + +// Digest hashes an explicit, versioned projection, never raw vendor documents. +func Digest(version string, value any) string { + b, _ := json.Marshal(value) + return fmt.Sprintf("sha256:%x", sha256.Sum256(append([]byte(version+"\n"), b...))) +} +func PhysicalID(role string, key []string) string { + return role + "-" + Digest("dpf-identity-v1", key)[7:] +} diff --git a/pkg/dpf/probe/collect.sh b/pkg/dpf/probe/collect.sh new file mode 100644 index 00000000..d5b93d71 --- /dev/null +++ b/pkg/dpf/probe/collect.sh @@ -0,0 +1,107 @@ +#!/bin/bash +# Copyright 2026 NVIDIA CORPORATION & AFFILIATES +# SPDX-License-Identifier: Apache-2.0 +# Fixed adapter for digest-pinned netshoot tools. No credentials, eval or repairs. +set -euo pipefail +umask 077 +request=$1 +node=$2 +mode=$3 +for tool in jq curl ip getent timeout date sort awk; do command -v "$tool" >/dev/null; done +[[ "$mode" == pod-network || "$mode" == node-network ]] +[[ -n "$node" && -n "$POD_UID" ]] +# Bound all commands, including DNS and network inspection, by the CLI deadline. +if [[ ${4:-} != bounded ]]; then + deadline=$(jq -r '.deadline | sub("\\.[0-9]+Z$"; "Z") | fromdateiso8601' "$request") + budget=$((deadline - $(date +%s))) + ((budget > 0 && budget <= 3600)) + exec timeout "$budget" /bin/bash "$0" "$request" "$node" "$mode" bounded +fi +ulimit -f 8192 +started=$(date -u +%Y-%m-%dT%H:%M:%SZ) +per_timeout=$(jq -r '.requestTimeout / 1000000000' "$request") +: > /scratch/results.jsonl +while IFS= read -r destination; do + host=$(jq -r '.hostname' <<<"$destination") + port=$(jq -r '.port' <<<"$destination") + verify_name=$(jq -r '.verifyName' <<<"$destination") + authority=$(jq -r '.authority' <<<"$destination") + url=$(jq -r '.url' <<<"$destination") + ca=$(jq -r '.caFile' <<<"$destination") + dns=resolved + if [[ $(jq -r '.literal' <<<"$destination") == true ]]; then + dns=not-applicable + addresses=$(jq -cn --arg host "$host" '[$host]') + else + # STREAM rows cover both families, deduplicated without a happy-eyeballs fallback. + # getent's lookup failure remains explicit unavailable evidence. + status=0 + timeout "$per_timeout" getent ahosts "$host" > /scratch/dns || status=$? + case "$status" in 0|2|124) ;; *) exit 1 ;; esac + if ((status != 0)); then + dns=unavailable + addresses='[]' + else + addresses=$(awk '$2 == "STREAM" { print $1 }' /scratch/dns | sort -u | jq -Rsc 'split("\n") | map(select(length > 0))') + if [[ $(jq 'length' <<<"$addresses") == 0 ]]; then dns=unavailable; fi + fi + fi + (($(jq 'length' <<<"$addresses") <= 32)) + : > /scratch/addresses.jsonl + while IFS= read -r address; do + resolved=$address + if [[ "$resolved" == *:* ]]; then resolved="[$resolved]"; fi + name=$verify_name + if [[ "$name" == *:* ]]; then name="[$name]"; fi + ca_args=() + if [[ -n "$ca" ]]; then ca_args=(--cacert "$ca"); fi + # Disable curlrc, proxies, redirects and all non-HTTPS protocols. Each IP + # is pinned while retaining the original HTTP authority and TLS identity. + status=0 + curl --disable --silent --show-error --noproxy '*' --proxy '' \ + --proto '=https' --proto-redir '=https' --tlsv1.2 --max-redirs 0 \ + --connect-timeout "$per_timeout" --max-time "$per_timeout" \ + --max-filesize 65536 --retry 0 --output /scratch/body \ + --write-out '%{json}' --resolve "$name:$port:$resolved" \ + --header "Host: $authority" "${ca_args[@]}" --url "$url" \ + > /scratch/curl.json 2>/scratch/curl.stderr || status=$? + # A tool/configuration failure must not masquerade as a failed network path. + case "$status" in 0|5|6|7|18|28|35|52|55|56|60|63|80|92) ;; *) exit 1 ;; esac + jq -e --argjson status "$status" '.exitcode == $status and (.time_connect | type == "number") and (.time_appconnect | type == "number")' /scratch/curl.json >/dev/null + tcp=unavailable + tls=unavailable + http=http-unavailable + if jq -e '.time_connect > 0' /scratch/curl.json >/dev/null; then tcp=reachable; fi + if jq -e '.time_appconnect > 0 and .ssl_verify_result == 0' /scratch/curl.json >/dev/null; then tls=verified; fi + code=$(jq '.response_code' /scratch/curl.json) + if [[ "$tls" == verified ]]; then + if ((code == 401 || code == 403)); then http=authentication-required + elif ((code >= 300 && code < 400)); then http=redirect-refused + elif ((code == 404)); then http=resource-unavailable + elif ((code >= 400)); then http=application-error + elif ((code >= 200 && code < 300)); then + http=invalid-response + if ((status == 0)) && jq -se 'length == 1 and (.[0] | type == "object")' /scratch/body >/dev/null 2>&1; then http=observed; fi + fi + fi + jq -cn --arg address "$address" --arg tcp "$tcp" --arg tls "$tls" --arg http "$http" --argjson code "$code" \ + '{address:$address,tcp:$tcp,tls:$tls,http:$http,httpStatus:$code}' >> /scratch/addresses.jsonl + done < <(jq -r '.[]' <<<"$addresses") + jq -cn --arg node "$node" --arg mode "$mode" --arg dns "$dns" \ + --arg id "$(jq -r '.id' <<<"$destination")" --slurpfile addresses /scratch/addresses.jsonl \ + '{nodeUID:$node,networkMode:$mode,destination:$id,dns:$dns,addresses:$addresses,evidenceRef:""}' >> /scratch/results.jsonl +done < <(jq -c '.destinations[]' "$request") +printf '[]' > /scratch/interfaces.json +printf '[]' > /scratch/routes.json +if [[ "$mode" == node-network ]]; then + ip -j -N address show > /scratch/links.json + ip -j -N -4 route show table all > /scratch/routes4.json + ip -j -N -6 route show table all > /scratch/routes6.json + jq '[.[] | {name:.ifname,index:.ifindex,mtu:.mtu,addresses:[.addr_info[] | "\(.local)/\(.prefixlen)"]}]' /scratch/links.json > /scratch/interfaces.json + jq -n --slurpfile links /scratch/links.json --slurpfile r4 /scratch/routes4.json --slurpfile r6 /scratch/routes6.json \ + '[($r4[0]+$r6[0])[] | . as $r | {destination:(.dst // "default"),gateway:(.gateway // ""),table:((.table // 254)|tonumber),interfaceIndex:([$links[0][] | select(.ifname == $r.dev) | .ifindex][0] // 0)}]' > /scratch/routes.json +fi +jq -cn --arg started "$started" --arg finished "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg node "$node" --arg pod "$POD_UID" --arg mode "$mode" --slurpfile request "$request" \ + --slurpfile results /scratch/results.jsonl --slurpfile interfaces /scratch/interfaces.json --slurpfile routes /scratch/routes.json \ + '{startedAt:$started,finishedAt:$finished,version:$request[0].version,runID:$request[0].runID,nodeUID:$node,podUID:$pod,networkMode:$mode,results:$results,interfaces:$interfaces[0],routes:$routes[0]}' diff --git a/pkg/dpf/probe/image_test.go b/pkg/dpf/probe/image_test.go new file mode 100644 index 00000000..2c776f70 --- /dev/null +++ b/pkg/dpf/probe/image_test.go @@ -0,0 +1,128 @@ +//go:build dpf_probe_image + +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package probe + +import ( + "context" + "encoding/json" + "encoding/pem" + "fmt" + "net" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// This explicit integration gate runs existing image tools without a cluster. +// DPF_PROBE_TEST_IMAGE must be the immutable catalog image under review. +var _ = Describe("published image runtime", func() { + It("collects both vantages with real DNS, per-IP TLS, HTTP and read-only routes", func() { + image := os.Getenv("DPF_PROBE_TEST_IMAGE") + Expect(image).To(ContainSubstring("@sha256:")) + platform := os.Getenv("DPF_PROBE_TEST_PLATFORM") + if platform == "" { + platform = "linux/amd64" + } + Expect(platform).To(BeElementOf("linux/amd64", "linux/arm64")) + server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/401": + w.WriteHeader(401) + case "/403": + w.WriteHeader(403) + case "/302": + w.Header().Set("Location", "https://must-not-follow.invalid") + w.WriteHeader(302) + case "/404": + w.WriteHeader(404) + case "/500": + w.WriteHeader(500) + case "/invalid": + fmt.Fprint(w, "not JSON") + case "/large": + fmt.Fprint(w, `{"value":"`) + fmt.Fprint(w, string(make([]byte, 70000))) + fmt.Fprint(w, `"}`) + case "/multi": + fmt.Fprint(w, "{}\n{}") + case "/scalar-object": + fmt.Fprint(w, "42\n{}") + default: + fmt.Fprint(w, `{"ok":true}`) + } + })) + listener, err := net.Listen("tcp", "0.0.0.0:0") + Expect(err).NotTo(HaveOccurred()) + server.Listener.Close() + server.Listener = listener + server.StartTLS() + defer server.Close() + port := listener.Addr().(*net.TCPAddr).Port + endpoint := fmt.Sprintf("https://host.docker.internal:%d", port) + ca := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw}) + req := Request{Version: ProtocolVersion, RunID: "local-image-check", Deadline: time.Now().Add(2 * time.Minute), RequestTimeout: 3 * time.Second, Destinations: []Destination{}} + for _, p := range []string{"/ok", "/401", "/403", "/302", "/404", "/500", "/invalid", "/large", "/multi", "/scalar-object"} { + req.Destinations = append(req.Destinations, Destination{ID: p, Endpoint: endpoint, Path: p, ServerName: "example.com", CA: ca}) + } + req.Destinations = append(req.Destinations, Destination{ID: "untrusted", Endpoint: endpoint, Path: "/ok", ServerName: "example.com"}, Destination{ID: "dns-missing", Endpoint: "https://does-not-exist.invalid", Path: "/version"}) + for _, mode := range []string{"pod-network", "node-network"} { + dir := GinkgoT().TempDir() + Expect(os.Chmod(dir, 0755)).To(Succeed()) + req.Deadline = time.Now().Add(2 * time.Minute) + files, err := requestFiles(req) + Expect(err).NotTo(HaveOccurred()) + for key, value := range files { + Expect(os.WriteFile(filepath.Join(dir, key), []byte(value), 0644)).To(Succeed()) + } + ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) + command := exec.CommandContext(ctx, "docker", "run", "--rm", "--platform", platform, "--user", "65532:65532", "--read-only", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--memory", "64m", "--env", "POD_UID=pod-local", "--mount", "type=bind,src="+dir+",dst=/request,readonly", "--tmpfs", "/scratch:rw,size=16777216,uid=65532,gid=65532,mode=0700", "--workdir", "/scratch", "--entrypoint", "/bin/bash", image, "/request/collect.sh", "/request/request.json", "node-local", mode) + output, err := command.CombinedOutput() + cancel() + GinkgoWriter.Printf("image %s %s output: %s\n", image, mode, output) + Expect(err).NotTo(HaveOccurred(), string(output)) + var result Response + Expect(json.Unmarshal(output, &result)).To(Succeed()) + Expect(ValidateResponse(result, req, "node-local", "pod-local", mode)).To(Succeed()) + expected := []string{"observed", "authentication-required", "authentication-required", "redirect-refused", "resource-unavailable", "application-error", "invalid-response", "invalid-response", "invalid-response", "invalid-response"} + for i, state := range expected { + Expect(result.Results[i].Addresses).NotTo(BeEmpty()) + verified := 0 + for _, a := range result.Results[i].Addresses { + if a.TLS == "verified" { + verified++ + Expect(a.TCP).To(Equal("reachable")) + Expect(a.HTTP).To(Equal(state)) + } else { + // Docker Desktop can advertise an unroutable IPv6 host + // alternative. Retain its failure; do not drop that IP. + Expect(a.HTTP).To(Equal("http-unavailable")) + } + } + Expect(verified).To(BeNumerically(">", 0)) + } + Expect(result.Results[10].Addresses).NotTo(BeEmpty()) + reachable := false + for _, a := range result.Results[10].Addresses { + reachable = reachable || a.TCP == "reachable" + Expect(a.TLS).To(Equal("unavailable")) + } + Expect(reachable).To(BeTrue()) + Expect(result.Results[11].DNS).To(Equal("unavailable")) + if mode == "node-network" { + Expect(result.Interfaces).NotTo(BeEmpty()) + Expect(result.Routes).NotTo(BeEmpty()) + } else { + Expect(result.Interfaces).To(BeEmpty()) + Expect(result.Routes).To(BeEmpty()) + } + } + }) +}) diff --git a/pkg/dpf/probe/probe_test.go b/pkg/dpf/probe/probe_test.go new file mode 100644 index 00000000..ef4f0a1e --- /dev/null +++ b/pkg/dpf/probe/probe_test.go @@ -0,0 +1,168 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package probe + +import ( + "context" + "fmt" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/client-go/rest" + k8stest "k8s.io/client-go/testing" + "testing" + "time" +) + +func TestProbe(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "DPF probe ownership and protocol") +} + +var _ = Describe("probe integrity", func() { + It("declares two network vantages without credentials or privileged mounts", func() { + node := inventory.Node{Name: "cp", UID: "node-uid"} + a := Pod("probe", "default", map[string]string{"run": "one"}, node, "pod-network", "repo@sha256:d", "request", nil, 60) + b := Pod("probe", "default", a.Labels, node, "node-network", "repo@sha256:d", "request", nil, 60) + Expect(a.Spec.HostNetwork).To(BeFalse()) + Expect(b.Spec.HostNetwork).To(BeTrue()) + Expect(b.Spec.DNSPolicy).To(Equal(corev1.DNSDefault)) + Expect(*a.Spec.AutomountServiceAccountToken).To(BeFalse()) + Expect(a.Spec.Volumes).To(HaveLen(2)) + Expect(a.Spec.Volumes[0].Secret).To(BeNil()) + Expect(*a.Spec.Containers[0].SecurityContext.AllowPrivilegeEscalation).To(BeFalse()) + }) + It("accepts harmless lifecycle changes but rejects admission and ownership mutations", func() { + p := Pod("p", "default", nil, inventory.Node{Name: "n", UID: "u"}, "pod-network", "repo@sha256:d", "request", nil, 60) + live := p.DeepCopy() + live.ResourceVersion = "2" + live.Spec.NodeName = "n" + live.Status.Phase = corev1.PodSucceeded + Expect(samePod(p, live)).To(BeTrue()) + live.Spec.HostNetwork = true + Expect(samePod(p, live)).To(BeFalse()) + live = p.DeepCopy() + live.OwnerReferences = []metav1.OwnerReference{{UID: "foreign"}} + Expect(samePod(p, live)).To(BeFalse()) + }) + It("rejects duplicate rows and invalid address/state combinations", func() { + req := Request{Version: ProtocolVersion, RunID: "run", Deadline: time.Now().Add(time.Minute), Destinations: []Destination{{ID: "a"}}} + response := Response{Version: ProtocolVersion, RunID: "run", NodeUID: "node", PodUID: "pod", NetworkMode: "pod-network", StartedAt: time.Now(), FinishedAt: time.Now(), Results: []inventory.ProbeResult{{NodeUID: "node", NetworkMode: "pod-network", Destination: "a", DNS: "resolved", Addresses: []inventory.ProbeAddress{{Address: "192.0.2.1", TCP: "reachable", TLS: "verified", HTTP: "authentication-required"}}}}} + Expect(ValidateResponse(response, req, "node", "pod", "pod-network")).To(Succeed()) + response.Results[0].Addresses[0].TCP = "unavailable" + Expect(ValidateResponse(response, req, "node", "pod", "pod-network")).NotTo(Succeed()) + response.Results = append(response.Results, response.Results[0]) + Expect(ValidateResponse(response, req, "node", "pod", "pod-network")).NotTo(Succeed()) + }) + It("rejects absent image assets before Kubernetes writes", func() { + client := fake.NewClientset() + runner := Runner{} + _, _, e := runner.Run(context.Background(), config.Request{}, discovery.ManagementClient{Client: client}, &inventory.Inventory{}) + Expect(e).To(HaveOccurred()) + Expect(client.Actions()).To(BeEmpty()) + }) + It("rejects request ConfigMap admission mutation and preserves changed data", func() { + client := fake.NewClientset(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "default"}}) + client.PrependReactor("create", "configmaps", func(a k8stest.Action) (bool, runtime.Object, error) { + desired := a.(k8stest.CreateAction).GetObject().(*corev1.ConfigMap).DeepCopy() + desired.UID = "owned" + desired.ResourceVersion = "1" + desired.Data["request.json"] = "redirected" + Expect(client.Tracker().Create(corev1.SchemeGroupVersion.WithResource("configmaps"), desired, "default")).To(Succeed()) + return true, desired, nil + }) + assets := Assets{Qualified: true, Repository: "repo/probe", ManifestDigest: "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", PlatformDigests: map[string]string{"amd64": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"}} + runner := Runner{Assets: assets, Journal: func([]Receipt) error { return nil }} + inv := &inventory.Inventory{Collection: inventory.Collection{RunID: "run"}, Management: &inventory.Management{Nodes: []inventory.Node{}}, Evidence: map[string]inventory.Evidence{}} + _, _, e := runner.Run(context.Background(), config.Request{Timeout: time.Minute, RequestTimeout: time.Second}, discovery.ManagementClient{Client: client, REST: &rest.Config{Host: "https://api.invalid"}, Namespace: "default"}, inv) + Expect(e).To(MatchError(ContainSubstring("admission changed"))) + for _, a := range client.Actions() { + Expect(a.GetResource().Resource).NotTo(Equal("pods")) + } + }) +}) + +var _ = Describe("probe failure regressions", func() { + It("clears a delete conflict after authoritative owned-object absence", func() { + client := fake.NewClientset(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "default"}}) + client.PrependReactor("create", "configmaps", func(a k8stest.Action) (bool, runtime.Object, error) { + cm := a.(k8stest.CreateAction).GetObject().(*corev1.ConfigMap).DeepCopy() + cm.UID = "owned" + cm.ResourceVersion = "1" + Expect(client.Tracker().Create(corev1.SchemeGroupVersion.WithResource("configmaps"), cm, "default")).To(Succeed()) + return true, cm, nil + }) + client.PrependReactor("delete", "configmaps", func(a k8stest.Action) (bool, runtime.Object, error) { + name := a.(k8stest.DeleteAction).GetName() + Expect(client.Tracker().Delete(corev1.SchemeGroupVersion.WithResource("configmaps"), "default", name)).To(Succeed()) + return true, nil, apierrors.NewConflict(corev1.Resource("configmaps"), name, fmt.Errorf("changed then removed")) + }) + runner := Runner{Assets: Assets{Qualified: true, Repository: "repo/probe", ManifestDigest: "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", PlatformDigests: map[string]string{"amd64": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"}}, Journal: func([]Receipt) error { return nil }} + inv := &inventory.Inventory{Collection: inventory.Collection{RunID: "conflict"}, Management: &inventory.Management{Nodes: []inventory.Node{{Name: "not-ready", UID: "node", Ready: false}}}} + _, cleanup, err := runner.Run(context.Background(), config.Request{Timeout: time.Minute, RequestTimeout: time.Second}, discovery.ManagementClient{Client: client, REST: &rest.Config{Host: "https://api.invalid"}, Namespace: "default"}, inv) + Expect(err).To(HaveOccurred()) + Expect(cleanup.State).To(Equal("complete")) + Expect(cleanup.Leftovers).To(BeEmpty()) + }) +}) + +var _ = Describe("existing tool image adapter", func() { + It("normalizes IPv6 and separate TLS identity without injecting commands", func() { + req := Request{Version: ProtocolVersion, RunID: "run", Deadline: time.Now().Add(time.Minute), RequestTimeout: time.Second, Destinations: []Destination{{ID: "dpu", Endpoint: "https://[2001:db8::1]:8443", Path: "/redfish/v1/", ServerName: "bmc.example.invalid"}}} + files, err := requestFiles(req) + Expect(err).NotTo(HaveOccurred()) + Expect(files["collect.sh"]).To(Equal(collectorScript)) + Expect(files["request.json"]).To(ContainSubstring(`"hostname":"2001:db8::1"`)) + Expect(files["request.json"]).To(ContainSubstring(`"literal":true`)) + Expect(files["request.json"]).To(ContainSubstring(`"url":"https://bmc.example.invalid:8443/redfish/v1/"`)) + Expect(files["request.json"]).NotTo(ContainSubstring("password")) + req.Destinations[0].ServerName = "bmc.invalid\r\nInjected: yes" + _, err = requestFiles(req) + Expect(err).To(HaveOccurred()) + }) + It("rejects duplicate scope, expired deadlines, bad public trust and oversized maps", func() { + req := Request{Version: ProtocolVersion, RunID: "run", Deadline: time.Now().Add(time.Minute), RequestTimeout: time.Second, Destinations: []Destination{{ID: "api", Endpoint: "https://api.invalid", Path: "/version"}}} + req.Destinations = append(req.Destinations, req.Destinations[0]) + _, err := requestFiles(req) + Expect(err).To(HaveOccurred()) + req.Destinations = req.Destinations[:1] + req.Destinations[0].CA = []byte("not a certificate") + _, err = requestFiles(req) + Expect(err).To(MatchError(ContainSubstring("public probe CA"))) + req.Destinations[0].CA = nil + req.Deadline = time.Now().Add(-time.Second) + _, err = requestFiles(req) + Expect(err).To(HaveOccurred()) + req.Deadline = time.Now().Add(time.Minute) + req.RunID = string(make([]byte, 900<<10)) + _, err = requestFiles(req) + Expect(err).To(MatchError(ContainSubstring("ConfigMap budget"))) + }) + It("does not activate a pinned image before runtime qualification", func() { + assets := Assets{Repository: "repo/probe", ManifestDigest: "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", PlatformDigests: map[string]string{"amd64": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"}, Qualified: false} + Expect(assets.Validate()).To(MatchError(ContainSubstring("runtime qualification"))) + assets.Qualified = true + Expect(assets.Validate()).To(Succeed()) + }) + It("binds the fixed adapter and scratch volume into admission invariants", func() { + pod := Pod("p", "default", nil, inventory.Node{Name: "n", UID: "u"}, "pod-network", "repo@sha256:d", "request", nil, 60) + Expect(pod.Spec.Containers[0].Command).To(Equal([]string{"/bin/bash", "/request/collect.sh"})) + Expect(pod.Spec.Containers[0].WorkingDir).To(Equal("/scratch")) + Expect(*pod.Spec.Containers[0].SecurityContext.ReadOnlyRootFilesystem).To(BeTrue()) + Expect(pod.Spec.Volumes[1].EmptyDir.Medium).To(Equal(corev1.StorageMediumMemory)) + changed := pod.DeepCopy() + changed.Spec.Containers[0].Command = []string{"/bin/bash", "-c", "curl bad.invalid"} + Expect(samePod(pod, changed)).To(BeFalse()) + changed = pod.DeepCopy() + changed.Spec.Volumes[1].EmptyDir = nil + changed.Spec.Volumes[1].HostPath = &corev1.HostPathVolumeSource{Path: "/"} + Expect(samePod(pod, changed)).To(BeFalse()) + }) +}) diff --git a/pkg/dpf/probe/protocol.go b/pkg/dpf/probe/protocol.go new file mode 100644 index 00000000..b26177ba --- /dev/null +++ b/pkg/dpf/probe/protocol.go @@ -0,0 +1,137 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package probe contains the public diagnostic protocol and Kubernetes runner. +package probe + +import ( + "crypto/x509" + _ "embed" + "encoding/json" + "fmt" + "net/netip" + "net/url" + "strconv" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +const ProtocolVersion = "v1alpha1" + +// collectorScript adapts the existing image's tools to our public result format. +// It is fixed release code, never site-supplied commands or a downloaded helper. +// +//go:embed collect.sh +var collectorScript string + +type Destination struct { + ID string `json:"id"` + Endpoint string `json:"endpoint"` + Path string `json:"path"` + ServerName string `json:"serverName,omitempty"` + CA []byte `json:"ca,omitempty"` +} +type Request struct { + RequestTimeout time.Duration `json:"requestTimeout"` + Version string `json:"version"` + RunID string `json:"runID"` + Deadline time.Time `json:"deadline"` + Destinations []Destination `json:"destinations"` +} +type Response struct { + StartedAt time.Time `json:"startedAt"` + FinishedAt time.Time `json:"finishedAt"` + Version string `json:"version"` + RunID string `json:"runID"` + NodeUID string `json:"nodeUID"` + PodUID string `json:"podUID"` + NetworkMode string `json:"networkMode"` + Results []inventory.ProbeResult `json:"results"` + Interfaces []inventory.Interface `json:"interfaces,omitempty"` + Routes []inventory.Route `json:"routes,omitempty"` +} + +type toolDestination struct { + ID string `json:"id"` + Hostname string `json:"hostname"` + Port string `json:"port"` + Literal bool `json:"literal"` + URL string `json:"url"` + Authority string `json:"authority"` + VerifyName string `json:"verifyName"` + CAFile string `json:"caFile"` +} + +// requestFiles resolves URL/trust coordinates in Go. The script consumes JSON +// values as quoted arguments; it never parses URLs or evaluates site input. +func requestFiles(request Request) (map[string]string, error) { + if request.Version != ProtocolVersion || request.RunID == "" || len(request.Destinations) == 0 || len(request.Destinations) > 257 || time.Until(request.Deadline) > time.Hour || time.Until(request.Deadline) <= 0 || request.RequestTimeout < time.Second || request.RequestTimeout > 2*time.Minute { + return nil, fmt.Errorf("invalid probe request scope/deadline") + } + files := map[string]string{"collect.sh": collectorScript} + destinations := []toolDestination{} + seen := map[string]bool{} + for i, destination := range request.Destinations { + u, err := url.Parse(destination.Endpoint) + if err != nil || u.Scheme != "https" || u.Hostname() == "" || u.User != nil || u.RawQuery != "" || u.Fragment != "" || u.Path != "" || strings.ContainsAny(u.Host, "\r\n") || !strings.HasPrefix(destination.Path, "/") || destination.ID == "" || seen[destination.ID] { + return nil, fmt.Errorf("invalid or duplicate probe destination") + } + seen[destination.ID] = true + port := u.Port() + if port == "" { + port = "443" + } + portNumber, err := strconv.Atoi(port) + if err != nil || portNumber < 1 || portNumber > 65535 { + return nil, fmt.Errorf("invalid destination port") + } + name := destination.ServerName + if name == "" { + name = u.Hostname() + } + if strings.ContainsAny(name, "\r\n /@?#%") { + return nil, fmt.Errorf("invalid TLS server name") + } + host := name + if strings.Contains(host, ":") { + if _, err := netip.ParseAddr(host); err != nil { + return nil, fmt.Errorf("invalid TLS server name") + } + host = "[" + host + "]" + } + probeURL := url.URL{Scheme: "https", Host: host + ":" + port, Path: destination.Path} + _, literalErr := netip.ParseAddr(u.Hostname()) + d := toolDestination{ID: destination.ID, Hostname: u.Hostname(), Port: port, Literal: literalErr == nil, URL: probeURL.String(), Authority: u.Host, VerifyName: name} + if len(destination.CA) > 0 { + if !x509.NewCertPool().AppendCertsFromPEM(destination.CA) { + return nil, fmt.Errorf("invalid public probe CA") + } + key := fmt.Sprintf("ca-%d.pem", i) + files[key] = string(destination.CA) + d.CAFile = "/request/" + key + } + destinations = append(destinations, d) + } + document := struct { + Version string `json:"version"` + RunID string `json:"runID"` + Deadline time.Time `json:"deadline"` + RequestTimeout time.Duration `json:"requestTimeout"` + Destinations []toolDestination `json:"destinations"` + }{request.Version, request.RunID, request.Deadline.UTC(), request.RequestTimeout, destinations} + data, err := json.Marshal(document) + if err != nil { + return nil, err + } + files["request.json"] = string(data) + size := 0 + for key, value := range files { + size += len(key) + len(value) + } + if size > 900<<10 { + return nil, fmt.Errorf("probe request exceeds ConfigMap budget") + } + return files, nil +} diff --git a/pkg/dpf/probe/runner.go b/pkg/dpf/probe/runner.go new file mode 100644 index 00000000..6c238f9c --- /dev/null +++ b/pkg/dpf/probe/runner.go @@ -0,0 +1,490 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package probe + +import ( + "context" + "encoding/hex" + "encoding/json" + "fmt" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/artifacts" + "io" + "net/netip" + "net/url" + "path" + "reflect" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" +) + +// Assets are release-owned. A verified published digest is still subject to +// controlled Kubernetes admission/runtime qualification before CLI activation. +type Assets struct { + Repository string + ManifestDigest string + PlatformDigests map[string]string + Qualified bool +} + +func (a Assets) Validate() error { + if a.Repository == "" || !validDigest(a.ManifestDigest) || len(a.PlatformDigests) == 0 { + return fmt.Errorf("qualified DPF probe assets are not installed in this build") + } + for _, d := range a.PlatformDigests { + if !validDigest(d) { + return fmt.Errorf("invalid platform image digest") + } + } + if !a.Qualified { + return fmt.Errorf("DPF probe image is pinned but Kubernetes runtime qualification is pending") + } + return nil +} +func validDigest(s string) bool { + if len(s) != 71 || !strings.HasPrefix(s, "sha256:") { + return false + } + _, err := hex.DecodeString(s[7:]) + return err == nil +} + +type Receipt struct { + Kind string `json:"kind"` + Namespace string `json:"namespace"` + Name string `json:"name"` + UID types.UID `json:"uid,omitempty"` +} +type Runner struct { + Assets Assets + Journal func([]Receipt) error +} + +func (runner Runner) Run(ctx context.Context, r config.Request, mg discovery.ManagementClient, inv *inventory.Inventory) (rows []inventory.ProbeResult, cleanup inventory.Cleanup, err error) { + cleanup = inventory.Cleanup{State: "complete", Leftovers: []string{}} + if err := runner.Assets.Validate(); err != nil { + return nil, cleanup, err + } + if runner.Journal == nil { + return nil, cleanup, fmt.Errorf("probe resource journal required") + } + ns, err := mg.Client.CoreV1().Namespaces().Get(ctx, mg.Namespace, metav1.GetOptions{}) + if err != nil || ns.DeletionTimestamp != nil { + return nil, cleanup, fmt.Errorf("probe namespace unavailable") + } + request := Request{RequestTimeout: r.RequestTimeout, Version: ProtocolVersion, RunID: inv.Collection.RunID, Deadline: time.Now().Add(r.Timeout), Destinations: []Destination{}} + api, _ := url.Parse(mg.REST.Host) + request.Destinations = append(request.Destinations, Destination{ID: "management-api", Endpoint: api.Scheme + "://" + api.Host, Path: path.Join("/", api.Path, "version"), ServerName: mg.REST.ServerName, CA: mg.REST.CAData}) + for _, t := range r.Targets { + if t.Role == "dpus" { + request.Destinations = append(request.Destinations, Destination{ID: inventory.Digest("probe-destination-v1", t.Endpoint), Endpoint: t.Endpoint, Path: "/redfish/v1/", CA: t.Access.PublicCA()}) + } + } + if deadline, ok := ctx.Deadline(); ok { + request.Deadline = deadline + } + files, err := requestFiles(request) + if err != nil { + return nil, cleanup, err + } + receipts := []Receipt{} + pods := map[string]*corev1.Pod{} + var configMap *corev1.ConfigMap + journal := func(kind, name string, uid types.UID) error { + receipts = append(receipts, Receipt{Kind: kind, Namespace: mg.Namespace, Name: name, UID: uid}) + return runner.Journal(receipts) + } + defer func() { + cleanCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 60*time.Second) + defer cancel() + for i := len(receipts) - 1; i >= 0; i-- { + receipt := receipts[i] + if receipt.UID == "" { + if receipt.Kind == "Pod" { + live, e := mg.Client.CoreV1().Pods(mg.Namespace).Get(cleanCtx, receipt.Name, metav1.GetOptions{}) + if e == nil && samePod(pods[receipt.Name], live) { + receipt.UID = live.UID + } + } else { + live, e := mg.Client.CoreV1().ConfigMaps(mg.Namespace).Get(cleanCtx, receipt.Name, metav1.GetOptions{}) + if e == nil && sameConfigMap(configMap, live) { + receipt.UID = live.UID + } + } + if receipt.UID == "" { + cleanup.State = "incomplete" + cleanup.Leftovers = append(cleanup.Leftovers, receipt.Kind+"/"+receipt.Name+": create outcome unknown") + continue + } + receipts[i] = receipt + _ = runner.Journal(receipts) + } + var cleanupErr error + for { + if cleanCtx.Err() != nil { + cleanupErr = cleanCtx.Err() + break + } + options := metav1.DeleteOptions{Preconditions: &metav1.Preconditions{UID: &receipt.UID}} + if receipt.Kind == "Pod" { + live, e := mg.Client.CoreV1().Pods(mg.Namespace).Get(cleanCtx, receipt.Name, metav1.GetOptions{}) + if apierrors.IsNotFound(e) { + cleanupErr = nil + break + } + if e != nil { + cleanupErr = e + break + } + if live.UID != receipt.UID { + cleanupErr = nil + break + } + if !samePod(pods[receipt.Name], live) { + cleanupErr = fmt.Errorf("probe ownership/spec changed") + break + } + options.Preconditions.ResourceVersion = &live.ResourceVersion + cleanupErr = mg.Client.CoreV1().Pods(mg.Namespace).Delete(cleanCtx, receipt.Name, options) + } else { + live, e := mg.Client.CoreV1().ConfigMaps(mg.Namespace).Get(cleanCtx, receipt.Name, metav1.GetOptions{}) + if apierrors.IsNotFound(e) { + cleanupErr = nil + break + } + if e != nil { + cleanupErr = e + break + } + if live.UID != receipt.UID { + cleanupErr = nil + break + } + if !sameConfigMap(configMap, live) { + cleanupErr = fmt.Errorf("request ownership/data changed") + break + } + options.Preconditions.ResourceVersion = &live.ResourceVersion + cleanupErr = mg.Client.CoreV1().ConfigMaps(mg.Namespace).Delete(cleanCtx, receipt.Name, options) + } + if apierrors.IsNotFound(cleanupErr) { + cleanupErr = nil + break + } + if cleanupErr != nil && !apierrors.IsConflict(cleanupErr) { + break + } + select { + case <-cleanCtx.Done(): + cleanupErr = cleanCtx.Err() + case <-time.After(100 * time.Millisecond): + } + if cleanupErr != nil && !apierrors.IsConflict(cleanupErr) { + break + } + } + if cleanupErr != nil { + cleanup.State = "incomplete" + cleanup.Leftovers = append(cleanup.Leftovers, receipt.Kind+"/"+receipt.Name) + } + } + if cleanup.State != "complete" && err == nil { + err = fmt.Errorf("probe cleanup incomplete; see resource journal") + } + }() + name := "l8k-dpf-" + strings.TrimPrefix(inventory.Digest("run", inv.Collection.RunID), "sha256:")[:16] + labels := map[string]string{"app.kubernetes.io/managed-by": "l8k-dpf-discovery", "l8k.nvidia.com/discovery-run": name} + if err = journal("ConfigMap", name, ""); err != nil { + return nil, cleanup, err + } + desired := &corev1.ConfigMap{Immutable: ptr(true), ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: mg.Namespace, Labels: labels}, Data: files} + configMap = desired + configMap, err = mg.Client.CoreV1().ConfigMaps(mg.Namespace).Create(ctx, desired, metav1.CreateOptions{}) + if err != nil { + configMap = desired + if createRejected(err) { + receipts = receipts[:len(receipts)-1] + } + return nil, cleanup, fmt.Errorf("probe ConfigMap create outcome unavailable; inspect journal") + } + receipts[len(receipts)-1].UID = configMap.UID + if !sameConfigMap(desired, configMap) { + configMap = desired + return nil, cleanup, fmt.Errorf("admission changed probe request data/ownership") + } + if err = runner.Journal(receipts); err != nil { + return nil, cleanup, err + } + for _, node := range inv.Management.Nodes { + if !node.Ready || node.Unschedulable || node.OS != "linux" || runner.Assets.PlatformDigests[node.Architecture] == "" { + return rows, cleanup, fmt.Errorf("management node cannot run the qualified probe") + } + for _, mode := range []string{"pod-network", "node-network"} { + podName := name + "-" + strings.TrimPrefix(inventory.Digest("node", node.UID+mode), "sha256:")[:12] + image := runner.Assets.Repository + if r.ImageRepository != "" { + image = r.ImageRepository + } + image += "@" + runner.Assets.ManifestDigest + desiredPod := Pod(podName, mg.Namespace, labels, node, mode, image, name, r.ImagePullSecrets, int64(r.Timeout/time.Second)) + if err = journal("Pod", podName, ""); err != nil { + return rows, cleanup, err + } + pods[podName] = desiredPod + created, createErr := mg.Client.CoreV1().Pods(mg.Namespace).Create(ctx, desiredPod, metav1.CreateOptions{}) + if createErr != nil { + if createRejected(createErr) { + receipts = receipts[:len(receipts)-1] + } + return rows, cleanup, fmt.Errorf("probe Pod create outcome unavailable; inspect journal") + } + receipts[len(receipts)-1].UID = created.UID + pods[podName] = created + if err = runner.Journal(receipts); err != nil { + return rows, cleanup, err + } + if !samePod(desiredPod, created) { + return rows, cleanup, fmt.Errorf("probe admission changed required invariants") + } + var live *corev1.Pod + for { + live, err = mg.Client.CoreV1().Pods(mg.Namespace).Get(ctx, podName, metav1.GetOptions{}) + if err != nil { + return rows, cleanup, fmt.Errorf("probe Pod inspection failed") + } + if !samePod(created, live) || live.UID != created.UID { + return rows, cleanup, fmt.Errorf("probe Pod identity/spec changed") + } + if live.Status.Phase == corev1.PodSucceeded { + break + } + if live.Status.Phase == corev1.PodFailed { + return rows, cleanup, fmt.Errorf("probe execution failed") + } + select { + case <-ctx.Done(): + return rows, cleanup, ctx.Err() + case <-time.After(200 * time.Millisecond): + } + } + currentNode, err := mg.Client.CoreV1().Nodes().Get(ctx, node.Name, metav1.GetOptions{}) + if err != nil || string(currentNode.UID) != node.UID || live.Spec.NodeName != node.Name { + return rows, cleanup, fmt.Errorf("probe node identity changed") + } + if len(live.Status.ContainerStatuses) != 1 || !strings.HasSuffix(live.Status.ContainerStatuses[0].ImageID, runner.Assets.PlatformDigests[node.Architecture]) { + return rows, cleanup, fmt.Errorf("probe pulled image digest differs from qualified asset") + } + stream, err := mg.Client.CoreV1().Pods(mg.Namespace).GetLogs(podName, &corev1.PodLogOptions{Container: "probe", LimitBytes: ptr(int64(4 << 20))}).Stream(ctx) + if err != nil { + return rows, cleanup, fmt.Errorf("probe log access failed") + } + body, readErr := io.ReadAll(io.LimitReader(stream, (4<<20)+1)) + stream.Close() + if readErr != nil || len(body) > 4<<20 { + return rows, cleanup, fmt.Errorf("probe result too large") + } + var result Response + if json.Unmarshal(body, &result) != nil || ValidateResponse(result, request, node.UID, string(live.UID), mode) != nil { + return rows, cleanup, fmt.Errorf("probe result binding/coverage invalid") + } + frozenMap, mapErr := mg.Client.CoreV1().ConfigMaps(mg.Namespace).Get(ctx, name, metav1.GetOptions{}) + if mapErr != nil || frozenMap.UID != configMap.UID || !sameConfigMap(configMap, frozenMap) { + return rows, cleanup, fmt.Errorf("probe request identity changed") + } + if mode == "node-network" { + for i := range inv.Management.Nodes { + if inv.Management.Nodes[i].UID == node.UID { + inv.Management.Nodes[i].Interfaces = result.Interfaces + inv.Management.Nodes[i].Routes = result.Routes + } + } + } + for _, row := range result.Results { + id := inventory.Digest("probe-evidence-v1", []string{node.UID, mode, row.Destination}) + row.EvidenceRef = id + inv.Evidence[id] = inventory.Evidence{Kind: "connectivity-probe", ObservedAt: result.FinishedAt, Vantage: node.UID + "/" + mode, ResourceURI: "pods/" + string(live.UID), Namespace: mg.Namespace, PodUID: string(live.UID), NodeUID: node.UID, ImageDigest: runner.Assets.PlatformDigests[node.Architecture], DNSPolicy: string(live.Spec.DNSPolicy), DestinationEndpoint: destinationEndpoint(request, row.Destination)} + rows = append(rows, row) + if row.DNS == "unavailable" || len(row.Addresses) == 0 { + return rows, cleanup, fmt.Errorf("required probe DNS coverage unavailable") + } + for _, a := range row.Addresses { + if a.TCP != "reachable" || a.TLS != "verified" { + return rows, cleanup, fmt.Errorf("required probe TCP/TLS path failed") + } + } + } + } + } + return rows, cleanup, nil +} +func ptr[T any](value T) *T { return &value } +func Pod(name, namespace string, labels map[string]string, node inventory.Node, mode, image, configMap string, secrets []string, seconds int64) *corev1.Pod { + container := corev1.Container{ + Name: "probe", Image: image, + Command: []string{"/bin/bash", "/request/collect.sh"}, + Args: []string{"/request/request.json", node.UID, mode}, + WorkingDir: "/scratch", + Env: []corev1.EnvVar{{Name: "POD_UID", ValueFrom: &corev1.EnvVarSource{FieldRef: &corev1.ObjectFieldSelector{FieldPath: "metadata.uid"}}}}, + SecurityContext: &corev1.SecurityContext{ + AllowPrivilegeEscalation: ptr(false), ReadOnlyRootFilesystem: ptr(true), + Capabilities: &corev1.Capabilities{Drop: []corev1.Capability{"ALL"}}, + }, + Resources: corev1.ResourceRequirements{ + Requests: corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("10m"), corev1.ResourceMemory: resource.MustParse("32Mi")}, + Limits: corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("100m"), corev1.ResourceMemory: resource.MustParse("64Mi")}, + }, + VolumeMounts: []corev1.VolumeMount{{Name: "request", MountPath: "/request", ReadOnly: true}, {Name: "scratch", MountPath: "/scratch"}}, + } + spec := corev1.PodSpec{ + RestartPolicy: corev1.RestartPolicyNever, AutomountServiceAccountToken: ptr(false), ActiveDeadlineSeconds: &seconds, + HostNetwork: mode == "node-network", DNSPolicy: corev1.DNSClusterFirst, + SecurityContext: &corev1.PodSecurityContext{ + RunAsNonRoot: ptr(true), RunAsUser: ptr(int64(65532)), + FSGroup: ptr(int64(65532)), + SeccompProfile: &corev1.SeccompProfile{Type: corev1.SeccompProfileTypeRuntimeDefault}, + }, + Affinity: &corev1.Affinity{NodeAffinity: &corev1.NodeAffinity{ + RequiredDuringSchedulingIgnoredDuringExecution: &corev1.NodeSelector{ + NodeSelectorTerms: []corev1.NodeSelectorTerm{{MatchFields: []corev1.NodeSelectorRequirement{{Key: "metadata.name", Operator: corev1.NodeSelectorOpIn, Values: []string{node.Name}}}}}, + }, + }}, + Tolerations: []corev1.Toleration{ + {Key: "node-role.kubernetes.io/control-plane", Operator: corev1.TolerationOpExists, Effect: corev1.TaintEffectNoSchedule}, + {Key: "node-role.kubernetes.io/master", Operator: corev1.TolerationOpExists, Effect: corev1.TaintEffectNoSchedule}, + }, + Containers: []corev1.Container{container}, + Volumes: []corev1.Volume{{Name: "request", VolumeSource: corev1.VolumeSource{ConfigMap: &corev1.ConfigMapVolumeSource{LocalObjectReference: corev1.LocalObjectReference{Name: configMap}}}}, {Name: "scratch", VolumeSource: corev1.VolumeSource{EmptyDir: &corev1.EmptyDirVolumeSource{Medium: corev1.StorageMediumMemory, SizeLimit: ptr(resource.MustParse("16Mi"))}}}}, + } + if mode == "node-network" { + spec.DNSPolicy = corev1.DNSDefault + } + for _, secret := range secrets { + spec.ImagePullSecrets = append(spec.ImagePullSecrets, corev1.LocalObjectReference{Name: secret}) + } + return &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace, Labels: labels}, Spec: spec} +} +func samePod(expected, actual *corev1.Pod) bool { + if expected == nil || actual == nil { + return false + } + a, b := expected.Spec.DeepCopy(), actual.Spec.DeepCopy() + // Only known harmless server/scheduler defaults are excluded from comparison. + normalize := func(s *corev1.PodSpec) { + s.NodeName = "" + tolerations := []corev1.Toleration{} + for _, t := range s.Tolerations { + standard := (t.Key == "node.kubernetes.io/not-ready" || t.Key == "node.kubernetes.io/unreachable") && t.Operator == corev1.TolerationOpExists && t.Effect == corev1.TaintEffectNoExecute && t.TolerationSeconds != nil && *t.TolerationSeconds == 300 + if !standard { + tolerations = append(tolerations, t) + } + } + s.Tolerations = tolerations + s.ServiceAccountName = "" + s.DeprecatedServiceAccount = "" + s.SchedulerName = "" + s.TerminationGracePeriodSeconds = nil + s.EnableServiceLinks = nil + s.PreemptionPolicy = nil + s.Priority = nil + + for i := range s.Containers { + s.Containers[i].ImagePullPolicy = "" + s.Containers[i].TerminationMessagePath = "" + s.Containers[i].TerminationMessagePolicy = "" + for j := range s.Containers[i].Env { + if src := s.Containers[i].Env[j].ValueFrom; src != nil && src.FieldRef != nil { + src.FieldRef.APIVersion = "" + } + } + } + for i := range s.Volumes { + if s.Volumes[i].ConfigMap != nil { + s.Volumes[i].ConfigMap.DefaultMode = nil + } + } + } + normalize(a) + normalize(b) + return reflect.DeepEqual(a, b) && reflect.DeepEqual(expected.Labels, actual.Labels) && reflect.DeepEqual(expected.OwnerReferences, actual.OwnerReferences) && reflect.DeepEqual(expected.Annotations, actual.Annotations) +} +func ValidateResponse(out Response, req Request, node, pod, mode string) error { + if out.Version != ProtocolVersion || out.RunID != req.RunID || out.NodeUID != node || out.PodUID != pod || out.NetworkMode != mode || len(out.Results) != len(req.Destinations) { + return fmt.Errorf("probe binding mismatch") + } + if out.StartedAt.IsZero() || out.FinishedAt.Before(out.StartedAt) || out.FinishedAt.After(req.Deadline.Add(time.Minute)) { + return fmt.Errorf("invalid probe time bounds") + } + expected := map[string]bool{} + for _, d := range req.Destinations { + expected[d.ID] = true + } + for _, r := range out.Results { + if !expected[r.Destination] || r.NodeUID != node || r.NetworkMode != mode { + return fmt.Errorf("unexpected or duplicate row") + } + if err := validateRow(r); err != nil { + return err + } + delete(expected, r.Destination) + } + return nil +} + +// FileJournal contains no auth material and records intended creates before dispatch. +func FileJournal(filename string) func([]Receipt) error { + return func(receipts []Receipt) error { + b, err := json.MarshalIndent(receipts, "", " ") + if err != nil { + return err + } + return artifacts.AtomicWrite(filename, b) + } +} + +func sameConfigMap(a, b *corev1.ConfigMap) bool { + return a != nil && b != nil && a.Immutable != nil && b.Immutable != nil && *a.Immutable && *b.Immutable && reflect.DeepEqual(a.Data, b.Data) && reflect.DeepEqual(a.BinaryData, b.BinaryData) && reflect.DeepEqual(a.Labels, b.Labels) && reflect.DeepEqual(a.OwnerReferences, b.OwnerReferences) && reflect.DeepEqual(a.Annotations, b.Annotations) +} +func destinationEndpoint(request Request, id string) string { + for _, d := range request.Destinations { + if d.ID == id { + return d.Endpoint + } + } + return "" +} +func validateRow(r inventory.ProbeResult) error { + if r.DNS != "resolved" && r.DNS != "not-applicable" && r.DNS != "unavailable" { + return fmt.Errorf("invalid DNS state") + } + if len(r.Addresses) > 32 { + return fmt.Errorf("address limit exceeded") + } + seen := map[string]bool{} + for _, a := range r.Addresses { + if _, err := netip.ParseAddr(a.Address); err != nil || seen[a.Address] { + return fmt.Errorf("invalid/duplicate IP address") + } + seen[a.Address] = true + if a.TCP != "reachable" && a.TCP != "unavailable" || a.TLS != "verified" && a.TLS != "unavailable" || a.TLS == "verified" && a.TCP != "reachable" { + return fmt.Errorf("invalid TCP/TLS state") + } + switch a.HTTP { + case "observed", "authentication-required", "redirect-refused", "resource-unavailable", "application-error", "http-unavailable", "invalid-response": + default: + return fmt.Errorf("invalid HTTP state") + } + } + return nil +} + +func createRejected(err error) bool { + return apierrors.IsForbidden(err) || apierrors.IsUnauthorized(err) || apierrors.IsAlreadyExists(err) || apierrors.IsInvalid(err) +} diff --git a/pkg/dpf/redfish/administrator.go b/pkg/dpf/redfish/administrator.go new file mode 100644 index 00000000..3b5c26d4 --- /dev/null +++ b/pkg/dpf/redfish/administrator.go @@ -0,0 +1,223 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package redfish + +import ( + "context" + "net/url" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +// administrator binds the owned session's privileges to a matching local account. +// A username or a role name alone is not privilege evidence. Account documents +// and arbitrary OEM privileges are never retained in inventory. +func (c *Client) administrator(ctx context.Context, root Document) inventory.Observation { + fail := func(code, message string) inventory.Observation { + return inventory.Observation{State: "failed", ReasonCode: code, Message: message, Tool: "redfish"} + } + unavailable := func(err error) inventory.Observation { + state := fail("BMC_ADMIN_UNVERIFIABLE", "The authenticated BMC account and its effective administrator privileges could not be verified ("+Code(err)+").") + switch Code(err) { + case "ACCESS_DENIED", "AUTHENTICATION_FAILED": + state = fail("BMC_ADMIN_ACCESS_DENIED", "The BMC denied access needed to verify the authenticated account and its administrator privileges.") + case "PASSWORD_CHANGE_REQUIRED": + state = fail("BMC_PASSWORD_CHANGE_REQUIRED", "The BMC requires a password change before account access; discovery does not change passwords.") + } + return state + } + if c.passwordChangeRequired { + return unavailable(&Error{Code: "PASSWORD_CHANGE_REQUIRED"}) + } + username, _ := c.access.Credentials() + session, err := c.Get(ctx, c.session) + if err != nil { + return unavailable(err) + } + if username == "" || Text(session, "UserName") != username { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC session does not identify the authenticated account consistently.") + } + service, err := c.accountLink(Link(root, "AccountService"), "/redfish/v1/AccountService") + if err != nil { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC does not advertise a trusted standard AccountService link.") + } + doc, err := c.Get(ctx, service) + if err != nil { + return unavailable(err) + } + // AccountService policy can change after login. It can contradict local + // authentication, but cannot prove the origin of this existing session. + mode := "" + if value, present := doc["LocalAccountAuth"]; present { + var valid bool + mode, valid = value.(string) + if !valid || (mode != "Enabled" && mode != "Disabled" && mode != "Fallback" && mode != "LocalFirst") { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC exposes an invalid local authentication mode.") + } + if mode == "Disabled" { + return fail("BMC_ADMIN_UNVERIFIABLE", "Local BMC account authentication is disabled; a same-named local account cannot verify this session's privileges.") + } + } + localSource := false + if source, present := session["UserAuthenticationSource"]; present && source != nil { + value, valid := source.(string) + if !valid { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC session exposes an invalid authentication source.") + } + if _, err := c.accountLink(value, service); err != nil { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC session does not identify the trusted local AccountService as its authentication source; external accounts are not verified.") + } + localSource = true + } + if (mode == "Fallback" || mode == "LocalFirst") && !localSource { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC authentication mode permits external accounts, but the session does not establish a local authentication source.") + } + accountsURI, err := c.accountLink(Link(doc, "Accounts"), service+"/Accounts") + if err != nil { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC does not advertise a trusted standard account collection.") + } + accounts, err := c.accounts(ctx, accountsURI, username) + if err != nil { + return unavailable(err) + } + var matches []Resource + for _, account := range accounts { + if Text(account.Document, "UserName") == username { + matches = append(matches, account) + } + } + if len(matches) != 1 { + return fail("BMC_ADMIN_UNVERIFIABLE", "The authenticated BMC username does not match exactly one local account; external or ambiguous accounts are not verified.") + } + account := matches[0] + passwordChange, passwordKnown := account.Document["PasswordChangeRequired"].(bool) + if passwordChange { + return unavailable(&Error{Code: "PASSWORD_CHANGE_REQUIRED"}) + } + enabled, enabledKnown := account.Document["Enabled"].(bool) + locked, lockedKnown := account.Document["Locked"].(bool) + if !passwordKnown || !enabledKnown || !lockedKnown { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC account does not expose valid enabled, locked and password-change state.") + } + if !enabled || locked { + return fail("BMC_ADMIN_ACCOUNT_RESTRICTED", "The authenticated BMC account is disabled or locked.") + } + roleID := Text(account.Document, "RoleId") + if roleID == "" || strings.ContainsAny(roleID, "/?#%") { + return fail("BMC_ADMIN_UNVERIFIABLE", "The authenticated BMC account does not expose a usable role identifier.") + } + if roles, present := session["Roles"]; present { + values, valid := roles.([]any) + if !valid || len(values) != 1 || values[0] != roleID { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC session roles do not match the authenticated account's assigned role.") + } + } else if !localSource { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC session exposes neither a matching effective role nor verified local authentication; username equality alone does not establish privileges.") + } + roleURI, err := c.accountLink(Link(Object(account.Document, "Links"), "Role"), service+"/Roles/"+roleID) + if err != nil { + return fail("BMC_ADMIN_UNVERIFIABLE", "The authenticated BMC account does not advertise a trusted link to its assigned role.") + } + role, err := c.Get(ctx, roleURI) + if err != nil { + return unavailable(err) + } + if Text(role, "Id") != roleID { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC role resource does not match the account's assigned role.") + } + assigned, ok := role["AssignedPrivileges"].([]any) + if !ok { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC role does not expose a valid standard privilege list.") + } + privileges := map[string]bool{} + for _, value := range assigned { + privilege, valid := value.(string) + if !valid || privilege == "" || privileges[privilege] { + return fail("BMC_ADMIN_UNVERIFIABLE", "The BMC role exposes a malformed standard privilege list.") + } + privileges[privilege] = true + } + // DMTF DSP0266: custom roles with the same privileges behave equivalently. + for _, privilege := range []string{"Login", "ConfigureManager", "ConfigureUsers", "ConfigureComponents", "ConfigureSelf"} { + if !privileges[privilege] { + return fail("BMC_ADMIN_REQUIRED", "The authenticated BMC account lacks the full standard Administrator privilege set.") + } + } + if c.passwordChangeRequired { + return unavailable(&Error{Code: "PASSWORD_CHANGE_REQUIRED"}) + } + return inventory.Observation{State: "observed", ReasonCode: "BMC_ADMIN_VERIFIED", Message: "The BMC session has verified standard Administrator privileges; its matching local account is enabled, unlocked and requires no password change.", Tool: "redfish"} +} + +// accounts selects an advertised username-addressed account without requiring +// permission to inspect unrelated accounts. Numeric/opaque account IDs use the +// complete bounded collection instead. Returned UserName is always verified by +// administrator; the URI spelling alone never establishes the account identity. +func (c *Client) accounts(ctx context.Context, uri, username string) ([]Resource, error) { + doc, err := c.Get(ctx, uri) + if err != nil { + return nil, err + } + if next, present := doc["Members@odata.nextLink"]; present { + if _, valid := next.(string); !valid { + return nil, &Error{Code: "INVALID_RESPONSE"} + } + } + members, valid := doc["Members"].([]any) + count, counted := doc["Members@odata.count"].(float64) + if !valid || !counted || count != float64(len(members)) || Text(doc, "Members@odata.nextLink") != "" { + return c.collection(ctx, uri, uri+"/") + } + ownURI := uri + "/" + url.PathEscape(username) + ownRequestURI := "" + seen := map[string]bool{} + for _, member := range members { + ref, requestURI, err := c.accountMember(Link(Document{"member": member}, "member"), uri+"/") + if err != nil { + return nil, err + } + if seen[ref] { + return nil, &Error{Code: "DUPLICATE_MEMBER"} + } + seen[ref] = true + if ref == ownURI { + ownRequestURI = requestURI + } + } + if ownRequestURI != "" { + account, err := c.Get(ctx, ownRequestURI) + if err != nil { + return nil, err + } + return []Resource{{URI: ownURI, Document: account}}, nil + } + return c.collection(ctx, uri, uri+"/") +} + +// accountLink constrains account reads to the advertised standard resource tree, +// in addition to the transport's same-origin and traversal protections. +func (c *Client) accountLink(ref, expected string) (string, error) { + uri, err := c.guard(ref) + if err != nil { + return "", err + } + if strings.TrimSuffix(uri, "/") != expected { + return "", &Error{Code: "UNSAFE_ACCOUNT_LINK"} + } + return expected, nil +} + +func passwordChangeMessage(doc Document) bool { + for _, container := range []Document{doc, Object(doc, "error")} { + messages, _ := container["@Message.ExtendedInfo"].([]any) + for _, item := range messages { + id := Text(Object(Document{"message": item}, "message"), "MessageId") + if strings.HasPrefix(id, "Base.") && strings.HasSuffix(id, ".PasswordChangeRequired") { + return true + } + } + } + return false +} diff --git a/pkg/dpf/redfish/administrator_test.go b/pkg/dpf/redfish/administrator_test.go new file mode 100644 index 00000000..2a4831ad --- /dev/null +++ b/pkg/dpf/redfish/administrator_test.go @@ -0,0 +1,364 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package redfish + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("DPU BMC administrator evidence", func() { + It("verifies the authenticated account and effective privileges without password or privilege mutations", func() { + Expect(os.Setenv("DPF_ADMIN_TEST_PASSWORD", "credential-canary")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_ADMIN_TEST_PASSWORD") + for _, scenario := range []struct { + name, expected string + session, service, login Document + omitRoles, customRole bool + }{ + {name: "administrator", expected: "BMC_ADMIN_VERIFIED"}, {name: "custom full role", expected: "BMC_ADMIN_VERIFIED"}, {name: "absolute links", expected: "BMC_ADMIN_VERIFIED"}, {name: "unrelated account unreadable", expected: "BMC_ADMIN_VERIFIED"}, {name: "slash-terminated account member", expected: "BMC_ADMIN_VERIFIED"}, {name: "session role absent", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "session password change", expected: "BMC_PASSWORD_CHANGE_REQUIRED"}, {name: "malformed pagination", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "late password change", expected: "BMC_PASSWORD_CHANGE_REQUIRED"}, + {name: "username admin read only", expected: "BMC_ADMIN_REQUIRED"}, {name: "operator", expected: "BMC_ADMIN_REQUIRED"}, {name: "OEM only", expected: "BMC_ADMIN_REQUIRED"}, + {name: "account service denied", expected: "BMC_ADMIN_ACCESS_DENIED"}, {name: "account member denied", expected: "BMC_ADMIN_ACCESS_DENIED"}, {name: "role denied", expected: "BMC_ADMIN_ACCESS_DENIED"}, + {name: "account service missing", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "missing user", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "duplicate user", expected: "BMC_ADMIN_UNVERIFIABLE"}, + {name: "malformed privileges", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "duplicate privileges", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "missing state", expected: "BMC_ADMIN_UNVERIFIABLE"}, + {name: "disabled", expected: "BMC_ADMIN_ACCOUNT_RESTRICTED"}, {name: "locked", expected: "BMC_ADMIN_ACCOUNT_RESTRICTED"}, {name: "password change", expected: "BMC_PASSWORD_CHANGE_REQUIRED"}, + {name: "login password change", expected: "BMC_PASSWORD_CHANGE_REQUIRED"}, {name: "denied password change", expected: "BMC_PASSWORD_CHANGE_REQUIRED"}, + {name: "session identity mismatch", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "session role mismatch", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "role identity mismatch", expected: "BMC_ADMIN_UNVERIFIABLE"}, + {name: "foreign service", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "foreign role", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "foreign account", expected: "BMC_ADMIN_UNVERIFIABLE"}, + {name: "wrong account tree", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "query role", expected: "BMC_ADMIN_UNVERIFIABLE"}, {name: "incomplete accounts", expected: "BMC_ADMIN_UNVERIFIABLE"}, + {name: "local enabled roleless", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"LocalAccountAuth": "Enabled"}}, + {name: "local enabled with LDAP roleless", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"LocalAccountAuth": "Enabled", "LDAP": Document{"ServiceEnabled": true}}}, + {name: "enabled snapshot after external login", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"LocalAccountAuth": "Enabled"}, login: Document{"Roles": []any{"ReadOnly"}, "UserAuthenticationSource": "ldaps://directory.example.invalid"}}, + {name: "enabled with effective session role", expected: "BMC_ADMIN_VERIFIED", service: Document{"LocalAccountAuth": "Enabled"}}, + {name: "enabled with explicit local source", expected: "BMC_ADMIN_VERIFIED", omitRoles: true, service: Document{"LocalAccountAuth": "Enabled"}, session: Document{"UserAuthenticationSource": "/redfish/v1/AccountService"}}, + {name: "explicit local source roleless", expected: "BMC_ADMIN_VERIFIED", omitRoles: true, session: Document{"UserAuthenticationSource": "/redfish/v1/AccountService"}}, + {name: "fallback explicit local source", expected: "BMC_ADMIN_VERIFIED", omitRoles: true, service: Document{"LocalAccountAuth": "Fallback"}, session: Document{"UserAuthenticationSource": "/redfish/v1/AccountService"}}, + {name: "local first custom explicit source", expected: "BMC_ADMIN_VERIFIED", omitRoles: true, customRole: true, service: Document{"LocalAccountAuth": "LocalFirst"}, session: Document{"UserAuthenticationSource": "/redfish/v1/AccountService"}}, + {name: "legacy LDAP certificate metadata", expected: "BMC_ADMIN_VERIFIED", service: Document{"LDAP": Document{"Certificates": Document{"@odata.id": "/redfish/v1/AccountService/LDAP/Certificates"}}}}, + {name: "unknown nullable source with session role", expected: "BMC_ADMIN_VERIFIED", session: Document{"UserAuthenticationSource": nil}}, + {name: "external local collision", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"LocalAccountAuth": "Disabled", "LDAP": Document{"ServiceEnabled": true, "RemoteRoleMapping": []any{Document{"RemoteUser": "site-maintainer", "LocalRole": "ReadOnly"}}}}}, + {name: "disabled local with session role", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Disabled"}}, + {name: "disabled contradictory local source", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Disabled"}, session: Document{"UserAuthenticationSource": "/redfish/v1/AccountService"}}, + {name: "fallback roleless", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"LocalAccountAuth": "Fallback"}}, + {name: "fallback with session role only", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Fallback"}}, + {name: "local first roleless", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"LocalAccountAuth": "LocalFirst"}}, + {name: "local first with session role only", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "LocalFirst"}}, + {name: "LDAP enabled missing session proof", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"LDAP": Document{"ServiceEnabled": true}}}, + {name: "LDAP unknown missing session proof", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"LDAP": Document{"Certificates": Document{}}}}, + {name: "additional providers missing session proof", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, service: Document{"AdditionalExternalAccountProviders": Document{"@odata.id": "/redfish/v1/AccountService/ExternalAccountProviders"}}}, + {name: "external authentication source", expected: "BMC_ADMIN_UNVERIFIABLE", session: Document{"UserAuthenticationSource": "ldaps://directory.example.invalid"}}, + {name: "enabled contradictory external source", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Enabled"}, session: Document{"UserAuthenticationSource": "ldaps://directory.example.invalid"}}, + {name: "wrong local authentication resource", expected: "BMC_ADMIN_UNVERIFIABLE", session: Document{"UserAuthenticationSource": "/redfish/v1/AccountService/Accounts/7"}}, + {name: "query authentication source", expected: "BMC_ADMIN_UNVERIFIABLE", session: Document{"UserAuthenticationSource": "/redfish/v1/AccountService?x=1"}}, + {name: "empty authentication source", expected: "BMC_ADMIN_UNVERIFIABLE", session: Document{"UserAuthenticationSource": ""}}, + {name: "boolean authentication source", expected: "BMC_ADMIN_UNVERIFIABLE", session: Document{"UserAuthenticationSource": true}}, + {name: "object authentication source", expected: "BMC_ADMIN_UNVERIFIABLE", session: Document{"UserAuthenticationSource": Document{"@odata.id": "/redfish/v1/AccountService"}}}, + {name: "null source roleless", expected: "BMC_ADMIN_UNVERIFIABLE", omitRoles: true, session: Document{"UserAuthenticationSource": nil}}, + {name: "unknown local mode", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Other"}}, + {name: "empty local mode", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": ""}}, + {name: "null local mode", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": nil}}, + {name: "boolean local mode", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": false}}, + {name: "array local mode", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": []any{"Enabled"}}}, + {name: "null session roles", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Enabled"}, session: Document{"Roles": nil}}, + {name: "empty session roles", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Enabled"}, session: Document{"Roles": []any{}}}, + {name: "null session role member", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Enabled"}, session: Document{"Roles": []any{nil}}}, + {name: "empty session role member", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Enabled"}, session: Document{"Roles": []any{""}}}, + {name: "duplicate session role members", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Enabled"}, session: Document{"Roles": []any{"Administrator", "Administrator"}}}, + {name: "malformed session roles", expected: "BMC_ADMIN_UNVERIFIABLE", service: Document{"LocalAccountAuth": "Enabled"}, session: Document{"Roles": "Administrator"}}, + } { + By(scenario.name) + writes, visited := []string{}, []string{} + username, roleID := "site-maintainer", "Administrator" + if scenario.name == "username admin read only" || scenario.name == "unrelated account unreadable" { + username = "admin" + } + if scenario.name == "custom full role" || scenario.customRole { + roleID = "SiteMaintainer" + } + link := func(uri string) Document { return Document{"@odata.id": uri} } + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + visited = append(visited, r.URL.Path) + if r.Method != http.MethodGet { + writes = append(writes, r.Method+" "+r.URL.Path) + } + var doc Document + switch r.URL.Path { + case "/redfish/v1/": + doc = Document{"Links": Document{"Sessions": link("/redfish/v1/SessionService/Sessions")}, "Systems": link("/redfish/v1/Systems"), "Chassis": link("/redfish/v1/Chassis"), "AccountService": link("/redfish/v1/AccountService")} + if scenario.name == "late password change" { + doc["UpdateService"] = link("/redfish/v1/UpdateService") + } + if scenario.name == "account service missing" { + delete(doc, "AccountService") + } + if scenario.name == "foreign service" { + doc["AccountService"] = link("https://foreign.invalid/redfish/v1/AccountService") + } + case "/redfish/v1/SessionService/Sessions": + w.Header().Set("X-Auth-Token", "token-canary") + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/owned") + w.WriteHeader(http.StatusCreated) + doc = Document{} + for key, value := range scenario.login { + doc[key] = value + } + if scenario.name == "login password change" { + doc["@Message.ExtendedInfo"] = []any{Document{"MessageId": "Base.1.18.PasswordChangeRequired", "Message": "credential-canary"}} + } + case "/redfish/v1/SessionService/Sessions/owned": + if r.Method == http.MethodDelete { + w.WriteHeader(http.StatusNoContent) + return + } + doc = Document{"UserName": username, "Roles": []any{roleID}} + if scenario.name == "session password change" { + doc["@Message.ExtendedInfo"] = []any{Document{"MessageId": "Base.1.18.PasswordChangeRequired"}} + } + if scenario.name == "session role absent" || scenario.omitRoles { + delete(doc, "Roles") + } + if scenario.name == "session identity mismatch" { + doc["UserName"] = "someone-else" + } + if scenario.name == "session role mismatch" { + doc["Roles"] = []any{"ReadOnly"} + } + for key, value := range scenario.session { + doc[key] = value + } + case "/redfish/v1/AccountService": + if scenario.name == "account service denied" || scenario.name == "denied password change" { + w.WriteHeader(http.StatusForbidden) + if scenario.name == "denied password change" { + json.NewEncoder(w).Encode(Document{"error": Document{"@Message.ExtendedInfo": []any{Document{"MessageId": "Base.1.18.PasswordChangeRequired", "Message": "credential-canary"}}}}) + } + return + } + doc = Document{"Accounts": link("/redfish/v1/AccountService/Accounts")} + for key, value := range scenario.service { + doc[key] = value + } + case "/redfish/v1/AccountService/Accounts": + if scenario.name == "slash-terminated account member" { + doc = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/AccountService/Accounts/site-maintainer/")}} + } else { + doc = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/AccountService/Accounts/7")}} + } + if scenario.name == "unrelated account unreadable" { + doc["Members"] = []any{link("/redfish/v1/AccountService/Accounts/root"), link("/redfish/v1/AccountService/Accounts/admin")} + doc["Members@odata.count"] = 2 + } + if scenario.name == "malformed pagination" { + doc["Members@odata.nextLink"] = false + } + if scenario.name == "duplicate user" { + doc["Members"] = []any{link("/redfish/v1/AccountService/Accounts/7"), link("/redfish/v1/AccountService/Accounts/8")} + doc["Members@odata.count"] = 2 + } + if scenario.name == "foreign account" { + doc["Members"] = []any{link("https://foreign.invalid/redfish/v1/AccountService/Accounts/7")} + } + if scenario.name == "wrong account tree" { + doc["Members"] = []any{link("/redfish/v1/Systems/unrelated")} + } + if scenario.name == "incomplete accounts" { + doc["Members@odata.count"] = 2 + } + case "/redfish/v1/AccountService/Accounts/root": + w.WriteHeader(http.StatusInternalServerError) + return + case "/redfish/v1/AccountService/Accounts/7", "/redfish/v1/AccountService/Accounts/8", "/redfish/v1/AccountService/Accounts/admin", "/redfish/v1/AccountService/Accounts/site-maintainer/": + if scenario.name == "account member denied" { + w.WriteHeader(http.StatusForbidden) + return + } + doc = Document{"UserName": username, "Enabled": true, "Locked": false, "PasswordChangeRequired": false, "RoleId": roleID, "Links": Document{"Role": link("/redfish/v1/AccountService/Roles/" + roleID)}} + switch scenario.name { + case "missing user": + doc["UserName"] = "other" + case "missing state": + delete(doc, "PasswordChangeRequired") + case "disabled": + doc["Enabled"] = false + case "locked": + doc["Locked"] = true + case "password change": + doc["PasswordChangeRequired"] = true + case "foreign role": + doc["Links"] = Document{"Role": link("https://foreign.invalid/redfish/v1/AccountService/Roles/" + roleID)} + case "query role": + doc["Links"] = Document{"Role": link("/redfish/v1/AccountService/Roles/" + roleID + "?$skip=1")} + } + case "/redfish/v1/AccountService/Roles/Administrator", "/redfish/v1/AccountService/Roles/SiteMaintainer": + if scenario.name == "role denied" { + w.WriteHeader(http.StatusForbidden) + return + } + doc = Document{"Id": roleID, "IsPredefined": roleID == "Administrator", "AssignedPrivileges": []any{"Login", "ConfigureManager", "ConfigureUsers", "ConfigureComponents", "ConfigureSelf"}} + switch scenario.name { + case "username admin read only", "operator", "OEM only": + doc["AssignedPrivileges"] = []any{"Login", "ConfigureSelf"} + doc["OemPrivileges"] = []any{"Administrator"} + case "malformed privileges": + doc["AssignedPrivileges"] = "Administrator" + case "duplicate privileges": + doc["AssignedPrivileges"] = []any{"Login", "Login", "ConfigureManager", "ConfigureUsers", "ConfigureComponents", "ConfigureSelf"} + case "role identity mismatch": + doc["Id"] = "OtherRole" + } + case "/redfish/v1/UpdateService": + w.WriteHeader(http.StatusForbidden) + json.NewEncoder(w).Encode(Document{"error": Document{"@Message.ExtendedInfo": []any{Document{"MessageId": "Base.1.18.PasswordChangeRequired", "Message": "credential-canary"}}}}) + return + case "/redfish/v1/Systems": + doc = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Systems/DPU")}} + case "/redfish/v1/Systems/DPU": + doc = Document{"UUID": "11111111-1111-4111-8111-111111111111", "Manufacturer": "NVIDIA", "Model": "BlueField-3"} + case "/redfish/v1/Chassis": + doc = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Chassis/Card")}} + case "/redfish/v1/Chassis/Card": + doc = Document{"Model": "BlueField-3", "SerialNumber": "CARD-A", "Links": Document{"ComputerSystems": []any{link("/redfish/v1/Systems/DPU")}}} + default: + w.WriteHeader(http.StatusNotFound) + return + } + if scenario.name == "absolute links" { + absoluteLinks(doc, "https://"+r.Host) + } + json.NewEncoder(w).Encode(doc) + })) + source := fmt.Sprintf("credentials:\n read: {username: '%s', passwordRef: {env: DPF_ADMIN_TEST_PASSWORD}}\ndiscovery:\n tls: {insecureSkipVerify: true}\n dpus: [{bmcEndpoint: '%s', credentialRef: read}]\n hosts: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: read}]\n probes: {mode: inventory}\n", username, server.URL) + input, err := config.Decode([]byte(source), filepath.Join(GinkgoT().TempDir(), "site.yaml")) + Expect(err).NotTo(HaveOccurred()) + request, err := config.Resolve(input) + Expect(err).NotTo(HaveOccurred()) + device, evidence, err := (Collector{}).Collect(context.Background(), request.Targets[0], time.Second) + expectNoRedfishEvidence(device, evidence) + Expect(err).NotTo(HaveOccurred()) + Expect(device.ObservationState).To(Equal("observed")) + Expect(device.SerialNumber).To(Equal("CARD-A")) + admin := device.Observations["bmc-administrator"] + Expect(admin.ReasonCode).To(Equal(scenario.expected)) + Expect(admin.Message).NotTo(BeEmpty()) + if scenario.expected == "BMC_ADMIN_VERIFIED" { + Expect(admin.State).To(Equal("observed")) + Expect(visited).To(ContainElement("/redfish/v1/AccountService/Roles/" + roleID)) + } else { + Expect(admin.State).To(Equal("failed")) + } + encoded, err := json.Marshal([]any{device, evidence}) + Expect(err).NotTo(HaveOccurred()) + Expect(string(encoded)).NotTo(ContainSubstring("credential-canary")) + Expect(string(encoded)).NotTo(ContainSubstring("token-canary")) + Expect(string(encoded)).NotTo(ContainSubstring(`"` + username + `"`)) + Expect(visited).NotTo(ContainElement("/redfish/v1/Systems/unrelated")) + Expect(visited).NotTo(ContainElement("/redfish/v1/AccountService/Accounts/root")) + Expect(writes).To(Equal([]string{"POST /redfish/v1/SessionService/Sessions", "DELETE /redfish/v1/SessionService/Sessions/owned"})) + server.Close() + } + }) +}) + +var _ = Describe("slash-terminated account links", func() { + It("canonicalizes the collection and direct or paginated member links before checking usernames", func() { + const collection = "/redfish/v1/AccountService/Accounts" + c := &Client{origin: "https://bmc.example.invalid"} + canonical, err := c.accountLink(collection+"/", collection) + Expect(err).NotTo(HaveOccurred()) + Expect(canonical).To(Equal(collection)) + + for _, scenario := range []struct { + name string + paged bool + }{ + {name: "direct username-addressed member"}, + {name: "member on later page", paged: true}, + } { + By(scenario.name) + var visited []string + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + visited = append(visited, r.URL.Path) + var doc Document + switch r.URL.Path { + case collection: + if scenario.paged && r.URL.Query().Get("$skip") == "1" { + doc = Document{"Members@odata.count": 2, "Members": []any{Document{"@odata.id": collection + "/7/"}}} + } else if scenario.paged { + doc = Document{"Members@odata.count": 2, "Members": []any{Document{"@odata.id": collection + "/8"}}, "Members@odata.nextLink": collection + "?$skip=1"} + } else { + doc = Document{"Members@odata.count": 1, "Members": []any{Document{"@odata.id": collection + "/site-maintainer/"}}} + } + case collection + "/8": + doc = Document{"UserName": "someone-else"} + case collection + "/7/", collection + "/site-maintainer/": + doc = Document{"UserName": "site-maintainer"} + default: + w.WriteHeader(http.StatusNotFound) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(doc) + })) + client := &Client{origin: server.URL, http: server.Client()} + accounts, err := client.accounts(context.Background(), collection, "site-maintainer") + Expect(err).NotTo(HaveOccurred()) + matches := 0 + for _, account := range accounts { + if Text(account.Document, "UserName") == "site-maintainer" { + matches++ + } + } + Expect(matches).To(Equal(1), "URI spelling must not replace authenticated username verification") + if scenario.paged { + Expect(visited).To(ContainElement(collection + "/7/")) + } else { + Expect(visited).To(ContainElement(collection + "/site-maintainer/")) + } + server.Close() + } + }) + + It("rejects duplicate member aliases and unsafe paths in direct and paginated lists", func() { + const collection = "/redfish/v1/AccountService/Accounts" + for _, scenario := range []struct { + name string + members []any + next bool + want string + }{ + {name: "duplicate terminal slash alias", members: []any{Document{"@odata.id": collection + "/7"}, Document{"@odata.id": collection + "/7/"}}, want: "DUPLICATE_MEMBER"}, + {name: "nested member path", members: []any{Document{"@odata.id": collection + "/7/child"}}, want: "UNSAFE_ACCOUNT_LINK"}, + {name: "query on member path", members: []any{Document{"@odata.id": collection + "/7/?$skip=1"}}, want: "UNSAFE_ACCOUNT_LINK"}, + {name: "double terminal slash", members: []any{Document{"@odata.id": collection + "/7//"}}, want: "UNSAFE_LINK"}, + {name: "duplicate aliases in paginated list", members: []any{Document{"@odata.id": collection + "/7"}, Document{"@odata.id": collection + "/7/"}}, next: true, want: "DUPLICATE_MEMBER"}, + } { + By(scenario.name) + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + doc := Document{"Members@odata.count": len(scenario.members), "Members": scenario.members} + if scenario.next && r.URL.Query().Get("$skip") == "" { + doc["Members@odata.count"] = len(scenario.members) + doc["Members@odata.nextLink"] = collection + "?$skip=1" + } else if scenario.next { + delete(doc, "Members@odata.nextLink") + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(doc) + })) + client := &Client{origin: server.URL, http: server.Client()} + _, err := client.accounts(context.Background(), collection, "site-maintainer") + Expect(Code(err)).To(Equal(scenario.want)) + server.Close() + } + }) +}) diff --git a/pkg/dpf/redfish/bluefield.go b/pkg/dpf/redfish/bluefield.go new file mode 100644 index 00000000..e8cfc671 --- /dev/null +++ b/pkg/dpf/redfish/bluefield.go @@ -0,0 +1,110 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package redfish + +import "strings" + +// b4240Service recognizes the two observed BF4 service shapes without deriving +// missing ComputerSystem properties from names, descriptions or URI spellings. +func b4240Service(root, system Document) bool { + if !strings.EqualFold(Text(root, "Vendor"), "NVIDIA") { + return false + } + product := Text(root, "Product") + if product != "B4240" && product != "BlueField-4" { + return false + } + for _, key := range []string{"Manufacturer", "Model", "SerialNumber", "UUID"} { + if value, present := system[key]; present && value != nil { + if _, valid := value.(string); !valid { + return false + } + } + } + maker, model := Text(system, "Manufacturer"), Text(system, "Model") + return (maker == "" || strings.EqualFold(maker, "NVIDIA")) && (model == "" || model == "BlueField-4") +} + +func (c *Client) linkedResource(doc Document, key, expected string) bool { + links, _ := Object(doc, "Links")[key].([]any) + for _, link := range links { + uri, err := c.guard(Link(Document{"resource": link}, "resource")) + if err == nil && uri == expected { + return true + } + } + return false +} + +// b4240Cards requires an exact board identity and explicit graph relationships. +// New firmware exposes the serial on the physical Card. Older firmware exposes +// it on the BMC Component, reciprocally contained by a serial-less physical Card. +// Both shapes retain the same serial-based host association and reject ambiguity. +func (c *Client) b4240Cards(system Resource, chassis []Resource) ([]Resource, []Resource) { + cards, witnesses := []Resource{}, []Resource{} + for _, ch := range chassis { + d := ch.Document + if Text(d, "Model") != "B4240" || !strings.EqualFold(Text(d, "Manufacturer"), "NVIDIA") || !ValidSerial(Text(d, "SerialNumber")) || !c.linkedToSystem(d, system.URI) { + continue + } + serial, systemSerial := strings.TrimSpace(Text(d, "SerialNumber")), strings.TrimSpace(Text(system.Document, "SerialNumber")) + if systemSerial != "" && (!ValidSerial(systemSerial) || serial != systemSerial) { + continue + } + switch Text(d, "ChassisType") { + case "Card": + // The observed physical-card shape also exposes the serial on the SoC. + if !ValidSerial(systemSerial) { + continue + } + cards = append(cards, ch) + case "Component": + if !c.linkedResource(system.Document, "Chassis", ch.URI) { + continue + } + parentURI, err := c.guard(Link(Object(d, "Links"), "ContainedBy")) + if err != nil { + continue + } + for _, parent := range chassis { + if parent.URI != parentURI || Text(parent.Document, "ChassisType") != "Card" || !c.linkedToSystem(parent.Document, system.URI) || !c.linkedResource(parent.Document, "Contains", ch.URI) { + continue + } + if !b4240Parent(parent.Document) { + continue + } + parentSerial := strings.TrimSpace(Text(parent.Document, "SerialNumber")) + if parentSerial != "" && (!ValidSerial(parentSerial) || parentSerial != serial) { + continue + } + cards = append(cards, ch) + witnesses = append(witnesses, parent) + } + } + } + return cards, witnesses +} + +// A serial-less legacy Card is an unknown witness, not permission to ignore +// contradictory or malformed identity metadata when the service does expose it. +func b4240Parent(parent Document) bool { + for _, key := range []string{"Manufacturer", "Model", "SerialNumber"} { + if value, present := parent[key]; present && value != nil { + if _, valid := value.(string); !valid { + return false + } + } + } + maker, model := Text(parent, "Manufacturer"), Text(parent, "Model") + return (maker == "" || strings.EqualFold(maker, "NVIDIA")) && (model == "" || model == "B4240") +} + +// Identity-bearing fields are reread, including the graph used for legacy BF4. +func cardIdentity(doc Document) Document { + out := Document{} + for _, key := range []string{"Model", "Manufacturer", "SerialNumber", "ChassisType", "Links"} { + out[key] = doc[key] + } + return out +} diff --git a/pkg/dpf/redfish/bluefield_test.go b/pkg/dpf/redfish/bluefield_test.go new file mode 100644 index 00000000..9fa2f77d --- /dev/null +++ b/pkg/dpf/redfish/bluefield_test.go @@ -0,0 +1,240 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package redfish + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("BF4 B4240 physical identity", func() { + It("accepts the two explicit firmware graphs and rejects unsupported or conflicting identity", func() { + Expect(os.Setenv("DPF_BF4_PASSWORD", "fixture-password")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_BF4_PASSWORD") + for _, scenario := range []struct { + name string + legacy bool + code string + }{ + {"physical card", false, ""}, + {"PSID System", false, ""}, {"PSID fallback", false, ""}, {"PSID matching", false, ""}, {"PSID conflict", false, ""}, {"PSID malformed", false, ""}, {"PSID late System", false, ""}, {"PSID late chassis", false, ""}, {"PSID missing", false, ""}, {"PSID reread denied", false, "ACCESS_DENIED"}, {"legacy component", true, ""}, {"nullable legacy metadata", true, ""}, + {"ambiguous cards", false, "CARD_IDENTITY_AMBIGUOUS"}, {"wrong serial", false, "CARD_IDENTITY_AMBIGUOUS"}, + {"wrong model", false, "CARD_IDENTITY_AMBIGUOUS"}, {"wrong manufacturer", false, "CARD_IDENTITY_AMBIGUOUS"}, + {"foreign system link", false, "CARD_IDENTITY_AMBIGUOUS"}, {"foreign containment", true, "CARD_IDENTITY_AMBIGUOUS"}, + {"missing containment", true, "CARD_IDENTITY_AMBIGUOUS"}, {"wrong system chassis", true, "CARD_IDENTITY_AMBIGUOUS"}, + {"graph changes", true, "IDENTITY_CHANGED"}, {"malformed system model", true, "DPU_IDENTITY_CONFLICT"}, {"malformed system serial", true, "DPU_IDENTITY_CONFLICT"}, + {"conflicting parent model", true, "CARD_IDENTITY_AMBIGUOUS"}, {"conflicting parent maker", true, "CARD_IDENTITY_AMBIGUOUS"}, {"malformed parent serial", true, "CARD_IDENTITY_AMBIGUOUS"}, {"malformed parent model", true, "CARD_IDENTITY_AMBIGUOUS"}, {"nullable parent metadata", true, ""}, {"conflicting parent serial", true, "CARD_IDENTITY_AMBIGUOUS"}, {"malformed parent maker", true, "CARD_IDENTITY_AMBIGUOUS"}, + } { + By(scenario.name) + link := func(uri string) Document { return Document{"@odata.id": uri} } + boardReads := 0 + systemReads, cardReads := 0, 0 + writes := []string{} + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + writes = append(writes, r.Method+" "+r.URL.Path) + } + var doc Document + switch r.URL.Path { + case "/redfish/v1/": + product := "BlueField-4" + if scenario.legacy { + product = "B4240" + } + doc = Document{"Vendor": "Nvidia", "Product": product, "Links": Document{"Sessions": link("/redfish/v1/SessionService/Sessions")}, "Systems": link("/redfish/v1/Systems"), "Chassis": link("/redfish/v1/Chassis")} + case "/redfish/v1/SessionService/Sessions": + w.Header().Set("X-Auth-Token", "fixture-token") + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/owned") + w.WriteHeader(http.StatusCreated) + doc = Document{} + case "/redfish/v1/SessionService/Sessions/owned": + if r.Method == http.MethodDelete { + w.WriteHeader(http.StatusNoContent) + return + } + doc = Document{"UserName": "reader"} + case "/redfish/v1/Systems": + doc = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Systems/DPU")}} + case "/redfish/v1/Systems/DPU": + systemReads++ + if scenario.name == "PSID reread denied" && systemReads > 1 { + w.WriteHeader(http.StatusForbidden) + return + } + doc = Document{"Manufacturer": "Nvidia", "Model": "BlueField-4", "SerialNumber": "CARD-A", "UUID": "11111111-1111-4111-8111-111111111111", "Links": Document{"Chassis": []any{link("/redfish/v1/Chassis/BMC")}}} + if strings.HasPrefix(scenario.name, "PSID ") && scenario.name != "PSID fallback" && scenario.name != "PSID missing" { + doc["AssetTag"] = "MT_0000001774" + } + if scenario.name == "PSID malformed" { + doc["AssetTag"] = false + } + if scenario.name == "PSID late System" && systemReads > 1 { + doc["AssetTag"] = "MT_0000001775" + } + if scenario.legacy { + delete(doc, "Manufacturer") + delete(doc, "Model") + delete(doc, "SerialNumber") + delete(doc, "UUID") + } + if scenario.name == "nullable legacy metadata" { + doc["Manufacturer"] = nil + doc["Model"] = nil + doc["SerialNumber"] = nil + doc["UUID"] = nil + } + if scenario.name == "malformed system model" { + doc["Model"] = false + } + if scenario.name == "malformed system serial" { + doc["SerialNumber"] = 123 + } + if scenario.name == "wrong system chassis" { + doc["Links"] = Document{"Chassis": []any{link("https://foreign.invalid/redfish/v1/Chassis/BMC")}} + } + case "/redfish/v1/Chassis": + members := []any{link("/redfish/v1/Chassis/Board"), link("/redfish/v1/Chassis/BMC")} + if scenario.name == "ambiguous cards" { + members = append(members, link("/redfish/v1/Chassis/Other")) + } + doc = Document{"Members@odata.count": len(members), "Members": members} + case "/redfish/v1/Chassis/Board", "/redfish/v1/Chassis/Other": + cardReads++ + doc = Document{"ChassisType": "Card", "Manufacturer": "Nvidia", "Model": "B4240", "SerialNumber": "CARD-A", "Links": Document{"ComputerSystems": []any{link("/redfish/v1/Systems/DPU")}, "Contains": []any{link("/redfish/v1/Chassis/BMC")}}} + if scenario.name == "PSID fallback" || scenario.name == "PSID matching" || scenario.name == "PSID late chassis" { + doc["AssetTag"] = "MT_0000001774" + } + if scenario.name == "PSID conflict" || scenario.name == "PSID late chassis" && cardReads > 1 { + doc["AssetTag"] = "MT_0000001775" + } + if scenario.legacy { + delete(doc, "Manufacturer") + delete(doc, "Model") + delete(doc, "SerialNumber") + } + if scenario.name == "conflicting parent serial" { + doc["SerialNumber"] = "CARD-B" + } + if scenario.name == "malformed parent maker" { + doc["Manufacturer"] = false + } + if scenario.name == "conflicting parent model" { + doc["Model"] = "ConnectX-8" + } + if scenario.name == "conflicting parent maker" { + doc["Manufacturer"] = "Other" + } + if scenario.name == "malformed parent serial" { + doc["SerialNumber"] = 123 + } + if scenario.name == "malformed parent model" { + doc["Model"] = true + } + if scenario.name == "nullable parent metadata" { + doc["Manufacturer"] = nil + doc["Model"] = nil + doc["SerialNumber"] = nil + } + if scenario.name == "missing containment" { + Object(doc, "Links")["Contains"] = []any{} + } + if scenario.name == "wrong serial" { + doc["SerialNumber"] = "CARD-B" + } + if scenario.name == "wrong model" { + doc["Model"] = "B4240X" + doc["Description"] = "NVIDIA BlueField-4 B4240 DPU" + } + if scenario.name == "wrong manufacturer" { + doc["Manufacturer"] = "Other" + } + if scenario.name == "foreign system link" { + Object(doc, "Links")["ComputerSystems"] = []any{link("https://foreign.invalid/redfish/v1/Systems/DPU")} + } + case "/redfish/v1/Chassis/BMC": + boardReads++ + model := "BlueField-4" + if scenario.legacy { + model = "B4240" + } + doc = Document{"ChassisType": "Component", "Manufacturer": "Nvidia", "Model": model, "SerialNumber": "CARD-A", "Links": Document{"ComputerSystems": []any{link("/redfish/v1/Systems/DPU")}}} + if scenario.legacy { + Object(doc, "Links")["ContainedBy"] = link("/redfish/v1/Chassis/Board") + } + if scenario.name == "foreign containment" { + Object(doc, "Links")["ContainedBy"] = link("https://foreign.invalid/redfish/v1/Chassis/Board") + } + if scenario.name == "graph changes" && boardReads > 1 { + Object(doc, "Links")["ContainedBy"] = link("/redfish/v1/Chassis/Other") + } + default: + w.WriteHeader(http.StatusNotFound) + return + } + json.NewEncoder(w).Encode(doc) + })) + seed := fmt.Sprintf("credentials:\n read: {username: reader, passwordRef: {env: DPF_BF4_PASSWORD}}\ndiscovery:\n tls: {insecureSkipVerify: true}\n dpus: [{bmcEndpoint: '%s', credentialRef: read}]\n hosts: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: read}]\n probes: {mode: inventory}\n", server.URL) + input, err := config.Decode([]byte(seed), filepath.Join(GinkgoT().TempDir(), "site.yaml")) + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(input) + Expect(err).NotTo(HaveOccurred()) + d, evidence, err := (Collector{}).Collect(context.Background(), req.Targets[0], time.Second) + expectNoRedfishEvidence(d, evidence) + if scenario.code == "" { + Expect(err).NotTo(HaveOccurred()) + Expect(d.ObservationState).To(Equal("observed")) + Expect(d.SerialNumber).To(Equal("CARD-A")) + Expect(d.Model).To(Equal("B4240")) + if strings.HasPrefix(scenario.name, "PSID ") { + expected := "PSID_OBSERVED" + switch scenario.name { + case "PSID conflict": + expected = "PSID_CONFLICT" + case "PSID malformed": + expected = "PSID_INVALID" + case "PSID late System", "PSID late chassis": + expected = "PSID_CHANGED" + case "PSID missing": + expected = "PSID_NOT_EXPOSED" + } + o := d.Observations["psid"] + Expect(o.ReasonCode).To(Equal(expected)) + Expect(d.Runtime).To(BeNil()) + Expect(systemReads).To(Equal(2)) + Expect(cardReads).To(Equal(2)) + if expected == "PSID_OBSERVED" { + Expect(o.Values).To(Equal([]string{"MT_0000001774"})) + + } else { + Expect(o.Values).To(BeEmpty()) + } + } + if scenario.legacy { + Expect(d.SystemUUID).To(BeEmpty()) + Expect(d.Manufacturer).To(BeEmpty()) + } + } else { + Expect(Code(err)).To(Equal(scenario.code)) + Expect(d.ObservationState).NotTo(Equal("observed")) + if scenario.name == "PSID reread denied" || scenario.name == "graph changes" { + Expect(d.Observations["psid"].ReasonCode).To(Equal("PSID_RECHECK_FAILED")) + Expect(d.Observations["psid"].Values).To(BeEmpty()) + } + } + Expect(writes).To(Equal([]string{"POST /redfish/v1/SessionService/Sessions", "DELETE /redfish/v1/SessionService/Sessions/owned"})) + server.Close() + } + }) +}) diff --git a/pkg/dpf/redfish/client.go b/pkg/dpf/redfish/client.go new file mode 100644 index 00000000..a3257c8c --- /dev/null +++ b/pkg/dpf/redfish/client.go @@ -0,0 +1,453 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package redfish implements a bounded GET/session-only Redfish transport. +package redfish + +import ( + "bytes" + "context" + "crypto/tls" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/url" + "path" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +type Document map[string]any + +// Error deliberately omits response bodies, headers and private credentials. +type Error struct { + Code string + Method string + URI string + Status int +} + +func (e *Error) Error() string { + return fmt.Sprintf("Redfish %s: %s %s (HTTP %d)", e.Code, e.Method, e.URI, e.Status) +} +func Code(err error) string { + var e *Error + if errors.As(err, &e) { + return e.Code + } + if errors.Is(err, context.Canceled) { + return "CANCELLED" + } + if errors.Is(err, context.DeadlineExceeded) { + return "BMC_TIMEOUT" + } + return "BMC_UNAVAILABLE" +} + +// Reader intentionally has no POST/PATCH/action method available to adapters. +type Reader interface { + Get(context.Context, string) (Document, error) + Collection(context.Context, string) ([]Resource, error) +} +type Resource struct { + URI string + Document Document +} +type Client struct { + label string + origin string + http *http.Client + access config.Access + token string + session string + sessionUncertain bool + passwordChangeRequired bool + reads int + bytes int +} + +func New(target config.Target, timeout time.Duration) *Client { + transport := http.DefaultTransport.(*http.Transport).Clone() + transport.Proxy = nil + transport.TLSClientConfig = target.Access.TLSConfig() + transport.TLSHandshakeTimeout = timeout + transport.ResponseHeaderTimeout = timeout + return &Client{label: target.DisplayName(), origin: target.Endpoint, access: target.Access, http: &http.Client{Transport: transport, Timeout: timeout, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}} +} + +// Resource categories carry useful request metadata without exposing account +// names, session IDs, opaque member identifiers or pagination tokens. +func requestCategory(ref string) string { + u, err := url.Parse(ref) + if err != nil || !strings.HasPrefix(u.Path, "/redfish/v1/") { + return "other" + } + first, _, _ := strings.Cut(strings.TrimPrefix(u.Path, "/redfish/v1/"), "/") + switch first { + case "": + return "service-root" + case "SessionService": + return "session-service" + case "AccountService": + return "account-service" + case "Systems": + return "systems" + case "Chassis": + return "chassis" + case "Managers": + return "managers" + case "UpdateService": + return "update-service" + default: + return "other" + } +} +func (c *Client) guard(ref string) (string, error) { + u, e := url.Parse(ref) + if e != nil { + return "", &Error{Code: "UNSAFE_LINK"} + } + if u.Host != "" && !u.IsAbs() { + return "", &Error{Code: "CROSS_ORIGIN_LINK"} + } + if u.IsAbs() { + o, err := config.Origin(u.Scheme + "://" + u.Host) + if err != nil || o != c.origin { + return "", &Error{Code: "CROSS_ORIGIN_LINK"} + } + } + if len(ref) > 2048 || u.User != nil || u.Fragment != "" || u.Opaque != "" || u.RawPath != "" || !strings.HasPrefix(u.Path, "/redfish/v1/") || strings.Contains(u.Path, "..") || path.Clean(u.Path) != strings.TrimSuffix(u.Path, "/") { + return "", &Error{Code: "UNSAFE_LINK"} + } + // Queries are accepted only as explicit pagination $skip/$skiptoken parameters. + for key := range u.Query() { + if key != "$skip" && key != "$skiptoken" { + return "", &Error{Code: "UNSAFE_QUERY"} + } + } + u.Scheme = "" + u.Host = "" + return u.String(), nil +} +func (c *Client) request(ctx context.Context, method, ref string, body []byte) (doc Document, headers http.Header, err error) { + started := time.Now() + status := 0 + logger := log.FromContext(ctx) + logger.V(2).Info("Redfish request started", "method", method, "resource", requestCategory(ref)) + defer func() { + reason := "" + if err != nil { + reason = Code(err) + } + logger.V(2).Info("Redfish request completed", "method", method, "resource", requestCategory(ref), "httpStatus", status, "reasonCode", reason, "duration", time.Since(started)) + var failure *Error + if errors.As(err, &failure) { + failure.Method = method + // Pagination tokens can carry opaque private state. Retain only the path. + failure.URI = strings.Split(failure.URI, "?")[0] + } + }() + uri, e := c.guard(ref) + if e != nil { + return nil, nil, e + } + req, e := http.NewRequestWithContext(ctx, method, c.origin+uri, bytes.NewReader(body)) + if e != nil { + return nil, nil, &Error{Code: "REQUEST_INVALID", URI: uri} + } + req.Header.Set("Accept", "application/json") + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + if c.token != "" { + req.Header.Set("X-Auth-Token", c.token) + } + response, e := c.http.Do(req) + if e != nil { + if ctx.Err() != nil { + return nil, nil, ctx.Err() + } + var timeout net.Error + if errors.As(e, &timeout) && timeout.Timeout() { + return nil, nil, &Error{Code: "BMC_TIMEOUT", URI: uri} + } + var verification *tls.CertificateVerificationError + if errors.As(e, &verification) { + return nil, nil, &Error{Code: "TLS_VERIFICATION_FAILED", URI: uri} + } + return nil, nil, &Error{Code: "TRANSPORT_FAILED", URI: uri} + } + defer response.Body.Close() + status = response.StatusCode + if response.StatusCode < 200 || response.StatusCode >= 300 { + code := "HTTP_FAILED" + if response.StatusCode == 401 { + code = "AUTHENTICATION_FAILED" + } + if response.StatusCode == 403 { + code = "ACCESS_DENIED" + // Inspect only the bounded standard message identifier; never retain bodies. + var failure Document + if b, readErr := io.ReadAll(io.LimitReader(response.Body, 64<<10)); readErr == nil && json.Unmarshal(b, &failure) == nil && passwordChangeMessage(failure) { + code = "PASSWORD_CHANGE_REQUIRED" + c.passwordChangeRequired = true + } + } + if response.StatusCode == 404 { + code = "RESOURCE_NOT_FOUND" + } + return nil, nil, &Error{Code: code, URI: uri, Status: response.StatusCode} + } + b, e := io.ReadAll(io.LimitReader(response.Body, (4<<20)+1)) + c.bytes += len(b) + if e != nil { + if ctx.Err() != nil { + return nil, nil, ctx.Err() + } + var timeout net.Error + if errors.As(e, &timeout) && timeout.Timeout() { + return nil, nil, &Error{Code: "BMC_TIMEOUT", URI: uri} + } + return nil, nil, &Error{Code: "RESPONSE_READ_FAILED", URI: uri} + } + if len(b) > 4<<20 || c.bytes > 32<<20 { + return nil, nil, &Error{Code: "RESPONSE_LIMIT", URI: uri} + } + var d Document + if len(b) != 0 { + if e := json.Unmarshal(b, &d); e != nil || d == nil { + return nil, nil, &Error{Code: "INVALID_RESPONSE", URI: uri} + } + } + if passwordChangeMessage(d) { + c.passwordChangeRequired = true + } + return d, response.Header.Clone(), nil +} +func (c *Client) Get(ctx context.Context, ref string) (Document, error) { + c.reads++ + if c.reads > 256 { + return nil, &Error{Code: "RESOURCE_LIMIT"} + } + for attempt := 0; attempt < 3; attempt++ { + d, _, err := c.request(ctx, http.MethodGet, ref, nil) + if err == nil { + return d, nil + } + var remote *Error + retry := errors.As(err, &remote) && (remote.Code == "TRANSPORT_FAILED" || remote.Code == "BMC_TIMEOUT" || remote.Status == 429 || remote.Status == 502 || remote.Status == 503 || remote.Status == 504) + if !retry || attempt == 2 { + return nil, err + } + timer := time.NewTimer(time.Duration(attempt+1) * 100 * time.Millisecond) + select { + case <-ctx.Done(): + timer.Stop() + return nil, ctx.Err() + case <-timer.C: + } + } + return nil, &Error{Code: "BMC_UNAVAILABLE"} +} +func (c *Client) Open(ctx context.Context) (Document, error) { + root, err := c.Get(ctx, "/redfish/v1/") + if err != nil { + return nil, err + } + // The public root advertises login even when SessionService itself requires + // authentication. Never require an authenticated GET before creating a session. + collection := Link(Object(root, "Links"), "Sessions") + if collection == "" { + service := Link(root, "SessionService") + if service == "" { + return nil, &Error{Code: "SESSION_AUTH_UNSUPPORTED"} + } + service, err = c.guard(service) + if err != nil { + return nil, err + } + if strings.TrimSuffix(service, "/") != "/redfish/v1/SessionService" { + return nil, &Error{Code: "SESSION_URI_UNSUPPORTED"} + } + sd, err := c.Get(ctx, service) + if err != nil { + return nil, err + } + collection = Link(sd, "Sessions") + } + if collection == "" { + return nil, &Error{Code: "SESSION_AUTH_UNSUPPORTED"} + } + collection, err = c.guard(collection) + if err != nil { + return nil, err + } + collectionBase := strings.TrimSuffix(collection, "/") + if collectionBase != "/redfish/v1/SessionService/Sessions" { + return nil, &Error{Code: "SESSION_URI_UNSUPPORTED"} + } + username, password := c.access.Credentials() + body, _ := json.Marshal(map[string]string{"UserName": username, "Password": password}) + // Never retry an ambiguous login. The token is never serialized; only a validated + // non-secret session receipt may be reported when cleanup fails. + c.sessionUncertain = true + _, header, err := c.request(ctx, http.MethodPost, collection, body) + if err != nil { + var failure *Error + if errors.As(err, &failure) && failure.Status >= 400 && failure.Status < 500 && failure.Status != 408 { + c.sessionUncertain = false + } + } + clear(body) + if err != nil { + return nil, err + } + token := header.Get("X-Auth-Token") + session, err := c.guard(header.Get("Location")) + sessionID := strings.TrimPrefix(strings.TrimSuffix(session, "/"), collectionBase+"/") + if err != nil || token == "" || !strings.HasPrefix(session, collectionBase+"/") || sessionID == "" || strings.Contains(sessionID, "/") || strings.Contains(session, "?") { + return nil, &Error{Code: "INVALID_SESSION_RECEIPT"} + } + c.token = token + c.session = session + c.sessionUncertain = false + return root, nil +} +func (c *Client) Close(ctx context.Context) (err error) { + started := time.Now() + attempted := c.session != "" || c.sessionUncertain + if attempted { + ui.FromContext(ctx).Info("[RUNNING] %s: Redfish session cleanup", c.label) + } + defer func() { + c.token = "" + c.http.CloseIdleConnections() + state, reason := "complete", "" + if err != nil { + state, reason = "incomplete", Code(err) + } + log.FromContext(ctx).V(1).Info("Redfish session cleanup completed", "state", state, "attempted", attempted, "reasonCode", reason, "duration", time.Since(started)) + if attempted { + if err != nil { + ui.FromContext(ctx).Info("[FAIL] %s: Redfish session cleanup (%s)", c.label, reason) + } else { + ui.FromContext(ctx).Info("[PASS] %s: Redfish session cleanup completed", c.label) + } + } + }() + if c.session != "" { + _, _, err := c.request(ctx, http.MethodDelete, c.session, nil) + return err + } + if c.sessionUncertain { + return &Error{Code: "SESSION_OUTCOME_UNKNOWN"} + } + return nil +} + +// Collection returns already read members alongside an error. Required consumers +// must reject the error; optional inventory may retain those members as partial evidence. +func (c *Client) Collection(ctx context.Context, ref string) ([]Resource, error) { + return c.collection(ctx, ref, "") +} + +// accountMember validates one advertised account member and returns its +// canonical identity path plus the guarded URI to use for the request. Only a +// single terminal slash is normalized for comparisons; reads preserve the +// exact advertised URI. +func (c *Client) accountMember(ref, memberBase string) (canonical, requestURI string, err error) { + link, err := c.guard(ref) + if err != nil { + return "", "", err + } + if !strings.HasPrefix(link, memberBase) || strings.Contains(link, "?") { + return "", "", &Error{Code: "UNSAFE_ACCOUNT_LINK"} + } + id := strings.TrimPrefix(link, memberBase) + id = strings.TrimSuffix(id, "/") + if id == "" || strings.Contains(id, "/") { + return "", "", &Error{Code: "UNSAFE_ACCOUNT_LINK"} + } + return memberBase + id, link, nil +} + +func (c *Client) collection(ctx context.Context, ref, memberBase string) ([]Resource, error) { + var members []Resource + seen := map[string]bool{} + count := -1 + initial, err := c.guard(ref) + if err != nil { + return members, err + } + base := strings.Split(initial, "?")[0] + for page := 0; ref != ""; page++ { + if page >= 32 || seen[ref] { + return members, &Error{Code: "PAGINATION_LIMIT"} + } + seen[ref] = true + normalized, err := c.guard(ref) + if err != nil || strings.Split(normalized, "?")[0] != base { + return members, &Error{Code: "UNSAFE_PAGINATION"} + } + d, err := c.Get(ctx, ref) + if err != nil { + return members, err + } + if n, ok := d["Members@odata.count"].(float64); ok { + if n < 0 || n != float64(int(n)) || count >= 0 && count != int(n) { + return members, &Error{Code: "COLLECTION_CHANGED"} + } + count = int(n) + } + list, ok := d["Members"].([]any) + if !ok { + return members, &Error{Code: "INCOMPLETE_COLLECTION"} + } + for _, item := range list { + link := Link(Document{"member": item}, "member") + var identity string + if memberBase != "" { + identity, link, err = c.accountMember(link, memberBase) + } else { + link, err = c.guard(link) + identity = link + } + if err != nil { + return members, err + } + if identity == "" || seen["member:"+identity] { + return members, &Error{Code: "DUPLICATE_MEMBER"} + } + seen["member:"+identity] = true + value, err := c.Get(ctx, link) + if err != nil { + return members, err + } + members = append(members, Resource{URI: link, Document: value}) + } + ref = Text(d, "Members@odata.nextLink") + } + if count < 0 || count != len(members) { + return members, &Error{Code: "INCOMPLETE_COLLECTION"} + } + return members, nil +} +func Text(d Document, k string) string { s, _ := d[k].(string); return s } +func Object(d Document, k string) Document { + switch v := d[k].(type) { + case Document: + return v + case map[string]any: + return Document(v) + default: + return nil + } +} +func Link(d Document, k string) string { return Text(Object(d, k), "@odata.id") } diff --git a/pkg/dpf/redfish/client_test.go b/pkg/dpf/redfish/client_test.go new file mode 100644 index 00000000..eee91998 --- /dev/null +++ b/pkg/dpf/redfish/client_test.go @@ -0,0 +1,214 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "encoding/json" + "fmt" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "net/http" + "net/http/httptest" + "os" + "sync/atomic" + "testing" + "time" +) + +func TestRedfish(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "DPF Redfish") } + +var _ = Describe("GET/session-only transport", func() { + It("logs in through public root session links without reading the protected service", func() { + for _, scenario := range []struct{ slash, absolute bool }{{false, false}, {true, false}, {false, true}, {true, true}} { + collection := "/redfish/v1/SessionService/Sessions" + receipt := collection + "/owned" + if scenario.slash { + collection += "/" + receipt += "/" + } + var methods []string + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + methods = append(methods, r.Method+" "+r.URL.Path) + w.Header().Set("Content-Type", "application/json") + prefix := "" + if scenario.absolute { + prefix = "https://" + r.Host + } + switch r.URL.Path { + case "/redfish/v1/": + json.NewEncoder(w).Encode(Document{"SessionService": Document{"@odata.id": "/redfish/v1/SessionService"}, "Links": Document{"Sessions": Document{"@odata.id": prefix + collection}}}) + case collection: + Expect(r.Method).To(Equal("POST")) + w.Header().Set("Location", prefix+receipt) + w.Header().Set("X-Auth-Token", "private-token") + w.WriteHeader(http.StatusCreated) + w.Write([]byte(`{}`)) + case receipt: + Expect(r.Method).To(Equal("DELETE")) + Expect(r.Header.Get("X-Auth-Token")).To(Equal("private-token")) + w.WriteHeader(http.StatusNoContent) + default: + w.WriteHeader(http.StatusUnauthorized) + } + })) + c := &Client{origin: s.URL, http: s.Client()} + _, err := c.Open(context.Background()) + Expect(err).NotTo(HaveOccurred()) + Expect(c.Close(context.Background())).To(Succeed()) + Expect(methods).To(Equal([]string{"GET /redfish/v1/", "POST " + collection, "DELETE " + receipt})) + s.Close() + } + }) + It("rejects foreign, OEM and query-bearing session links without attempting login", func() { + for _, link := range []string{"https://foreign.invalid/redfish/v1/SessionService/Sessions", "/redfish/v1/Oem/Sessions", "/redfish/v1/SessionService/Sessions?$skip=1"} { + var writes atomic.Int32 + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + writes.Add(1) + } + json.NewEncoder(w).Encode(Document{"Links": Document{"Sessions": Document{"@odata.id": link}}}) + })) + c := &Client{origin: s.URL, http: s.Client()} + _, err := c.Open(context.Background()) + Expect(err).To(HaveOccurred()) + Expect(writes.Load()).To(BeZero()) + Expect(c.Close(context.Background())).To(Succeed()) + s.Close() + } + }) + It("does not accept the session collection itself as a cleanup receipt", func() { + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodPost { + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/") + w.Header().Set("X-Auth-Token", "private-token") + w.WriteHeader(http.StatusCreated) + w.Write([]byte(`{}`)) + return + } + Expect(r.Method).NotTo(Equal(http.MethodDelete)) + json.NewEncoder(w).Encode(Document{"Links": Document{"Sessions": Document{"@odata.id": "/redfish/v1/SessionService/Sessions/"}}}) + })) + defer s.Close() + c := &Client{origin: s.URL, http: s.Client()} + _, err := c.Open(context.Background()) + Expect(Code(err)).To(Equal("INVALID_SESSION_RECEIPT")) + Expect(Code(c.Close(context.Background()))).To(Equal("SESSION_OUTCOME_UNKNOWN")) + }) + It("requires explicit insecure opt-in for an untrusted server and still rejects foreign origins", func() { + var requests atomic.Int32 + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests.Add(1) + w.Write([]byte(`{"RedfishVersion":"1.0.0"}`)) + })) + defer s.Close() + Expect(os.Setenv("DPF_TLS_TEST_PASSWORD", "private-password")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_TLS_TEST_PASSWORD") + for _, policy := range []string{"", ", tls: {insecureSkipVerify: false}", ", tls: {insecureSkipVerify: true}"} { + source := fmt.Sprintf("management:\n kubeconfigRef: {file: unused}\ncredentials:\n read: {username: reader, passwordRef: {env: DPF_TLS_TEST_PASSWORD}}\ndiscovery:\n dpus: [{bmcEndpoint: '%s', credentialRef: read%s}]\n hosts: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: read}]\n", s.URL, policy) + in, err := config.Decode([]byte(source), "/site/input.yaml") + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(in) + Expect(err).NotTo(HaveOccurred()) + c := New(req.Targets[0], time.Second) + _, err = c.Get(context.Background(), "/redfish/v1/") + if req.Targets[0].Access.InsecureSkipVerify() { + Expect(err).NotTo(HaveOccurred()) + } else { + Expect(Code(err)).To(Equal("TLS_VERIFICATION_FAILED")) + Expect(requests.Load()).To(BeZero()) + } + _, err = c.Get(context.Background(), "https://foreign.invalid/redfish/v1/") + Expect(Code(err)).To(Equal("CROSS_ORIGIN_LINK")) + } + Expect(requests.Load()).To(Equal(int32(1))) + }) + It("allows the service root and logs out exactly its own session", func() { + var methods []string + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + methods = append(methods, r.Method+" "+r.URL.Path) + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/redfish/v1/": + json.NewEncoder(w).Encode(Document{"SessionService": Document{"@odata.id": "/redfish/v1/SessionService"}}) + case "/redfish/v1/SessionService": + json.NewEncoder(w).Encode(Document{"Sessions": Document{"@odata.id": "/redfish/v1/SessionService/Sessions"}}) + case "/redfish/v1/SessionService/Sessions": + Expect(r.Method).To(Equal("POST")) + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/own") + w.Header().Set("X-Auth-Token", "private-token") + w.WriteHeader(201) + w.Write([]byte(`{}`)) + case "/redfish/v1/SessionService/Sessions/own": + Expect(r.Method).To(Equal("DELETE")) + Expect(r.Header.Get("X-Auth-Token")).To(Equal("private-token")) + w.WriteHeader(204) + default: + w.WriteHeader(404) + } + })) + defer s.Close() + c := &Client{origin: s.URL, http: s.Client()} + _, e := c.Open(context.Background()) + Expect(e).NotTo(HaveOccurred()) + Expect(c.Close(context.Background())).To(Succeed()) + Expect(methods).To(Equal([]string{"GET /redfish/v1/", "GET /redfish/v1/SessionService", "POST /redfish/v1/SessionService/Sessions", "DELETE /redfish/v1/SessionService/Sessions/own"})) + }) + It("rejects unsafe links before credentials leave the process", func() { + c := &Client{origin: "https://example.invalid"} + for _, u := range []string{"https://elsewhere.invalid/redfish/v1/Systems", "http://example.invalid/redfish/v1/Systems", "/redfish/v1/%2e%2e/Accounts", "/redfish/v1/../Accounts", "/redfish/v1/Systems?secret=1", "//elsewhere.invalid/redfish/v1/Systems"} { + _, e := c.guard(u) + Expect(e).To(HaveOccurred()) + } + }) + It("follows only bounded same-collection pages and checks count", func() { + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/redfish/v1/Systems": + if r.URL.RawQuery == "" { + json.NewEncoder(w).Encode(Document{"Members@odata.count": 2, "Members": []any{Document{"@odata.id": "/redfish/v1/Systems/a"}}, "Members@odata.nextLink": "/redfish/v1/Systems?$skip=1"}) + } else { + json.NewEncoder(w).Encode(Document{"Members@odata.count": 2, "Members": []any{Document{"@odata.id": "/redfish/v1/Systems/b"}}}) + } + default: + w.Write([]byte(`{"UUID":"example"}`)) + } + })) + defer s.Close() + c := &Client{origin: s.URL, http: s.Client()} + members, e := c.Collection(context.Background(), "/redfish/v1/Systems") + Expect(e).NotTo(HaveOccurred()) + Expect(members).To(HaveLen(2)) + }) + It("never retries authorization failures or prints response secrets", func() { + calls := 0 + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls++ + w.WriteHeader(401) + w.Write([]byte("private-password")) + })) + defer s.Close() + c := &Client{origin: s.URL, http: s.Client()} + _, e := c.Get(context.Background(), "/redfish/v1/Systems") + Expect(e).To(HaveOccurred()) + Expect(e.Error()).NotTo(ContainSubstring("private-password")) + Expect(calls).To(Equal(1)) + }) + It("stops read retries at cancellation", func() { + ctx, cancel := context.WithCancel(context.Background()) + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { cancel(); w.WriteHeader(503) })) + defer s.Close() + c := &Client{origin: s.URL, http: s.Client()} + start := time.Now() + _, e := c.Get(ctx, "/redfish/v1/Systems") + Expect(e).To(HaveOccurred()) + Expect(time.Since(start)).To(BeNumerically("<", time.Second)) + }) + It("rejects placeholder physical serials", func() { + for _, v := range []string{"", "unknown", "N/A", "00000", "to be filled by o.e.m."} { + Expect(ValidSerial(v)).To(BeFalse()) + } + Expect(ValidSerial(" 000ABC-01 ")).To(BeTrue()) + }) +}) diff --git a/pkg/dpf/redfish/diagnostics.go b/pkg/dpf/redfish/diagnostics.go new file mode 100644 index 00000000..564da2d0 --- /dev/null +++ b/pkg/dpf/redfish/diagnostics.go @@ -0,0 +1,72 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import "errors" + +// Diagnostic contains only bounded public request metadata, never headers or bodies. +type Diagnostic struct { + Code string + Method string + URI string + Status int + Stage string +} + +// ObservationError retains independent observation and exact-session cleanup errors. +// SessionURI is a validated resource receipt, not a session token. +type ObservationError struct { + Primary error + Cleanup error + SessionURI string +} + +func (e *ObservationError) Error() string { + if e.Primary != nil { + return e.Primary.Error() + "; BMC session cleanup incomplete" + } + return "BMC session cleanup incomplete: " + e.Cleanup.Error() +} +func (e *ObservationError) Unwrap() error { + if e.Primary != nil { + return e.Primary + } + return e.Cleanup +} + +// Diagnostics preserves a failed observation even when cleanup also fails. +func Diagnostics(err error) []Diagnostic { + if err == nil { + return nil + } + var result []Diagnostic + appendError := func(err error, stage string) { + if err == nil { + return + } + d := Diagnostic{Code: Code(err), Stage: stage} + var remote *Error + if errors.As(err, &remote) { + d.Method, d.URI, d.Status = remote.Method, remote.URI, remote.Status + } + result = append(result, d) + } + var observation *ObservationError + if errors.As(err, &observation) { + appendError(observation.Primary, "observation") + appendError(observation.Cleanup, "cleanup") + } else { + appendError(err, "observation") + } + return result +} + +// CleanupReceipt returns only the exact validated receipt for a failed cleanup. +// An empty result means that no usable receipt is available. +func CleanupReceipt(err error) string { + var observation *ObservationError + if errors.As(err, &observation) { + return observation.SessionURI + } + return "" +} diff --git a/pkg/dpf/redfish/diagnostics_test.go b/pkg/dpf/redfish/diagnostics_test.go new file mode 100644 index 00000000..c5492295 --- /dev/null +++ b/pkg/dpf/redfish/diagnostics_test.go @@ -0,0 +1,73 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("observation and cleanup diagnostics", func() { + It("retains independent forbidden read and DELETE metadata with the exact recovery receipt", func() { + var writes []string + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + writes = append(writes, r.Method+" "+r.URL.Path) + } + switch r.URL.Path { + case "/redfish/v1/": + json.NewEncoder(w).Encode(Document{"Links": Document{"Sessions": Document{"@odata.id": "/redfish/v1/SessionService/Sessions"}}, "Systems": Document{"@odata.id": "/redfish/v1/Systems"}}) + case "/redfish/v1/SessionService/Sessions": + w.Header().Set("X-Auth-Token", "secret-token") + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/owned") + w.WriteHeader(http.StatusCreated) + w.Write([]byte(`{}`)) + default: + w.WriteHeader(http.StatusForbidden) + } + })) + defer s.Close() + Expect(os.Setenv("DPF_DIAGNOSTIC_PASSWORD", "secret-password")).To(Succeed()) + defer os.Unsetenv("DPF_DIAGNOSTIC_PASSWORD") + source := fmt.Sprintf("management:\n kubeconfigRef: {file: unused}\ncredentials:\n read: {username: reader, passwordRef: {env: DPF_DIAGNOSTIC_PASSWORD}}\ndiscovery:\n tls: {insecureSkipVerify: true}\n dpus: [{bmcEndpoint: '%s', credentialRef: read}]\n hosts: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: read}]\n", s.URL) + in, err := config.Decode([]byte(source), "/site/input.yaml") + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(in) + Expect(err).NotTo(HaveOccurred()) + device, _, err := (Collector{}).Collect(context.Background(), req.Targets[0], time.Second) + Expect(err).To(HaveOccurred()) + Expect(device.Authentication.PasswordRead).NotTo(Equal("verified")) + Expect(device.ReasonCode).To(Equal("CLEANUP_INCOMPLETE")) + Expect(Code(err)).To(Equal("ACCESS_DENIED")) + Expect(Diagnostics(err)).To(Equal([]Diagnostic{ + {Code: "ACCESS_DENIED", Method: "GET", URI: "/redfish/v1/Systems", Status: 403, Stage: "observation"}, + {Code: "ACCESS_DENIED", Method: "DELETE", URI: "/redfish/v1/SessionService/Sessions/owned", Status: 403, Stage: "cleanup"}, + })) + Expect(CleanupReceipt(err)).To(Equal("/redfish/v1/SessionService/Sessions/owned")) + Expect(err.Error()).NotTo(ContainSubstring("secret-")) + Expect(writes).To(Equal([]string{"POST /redfish/v1/SessionService/Sessions", "DELETE /redfish/v1/SessionService/Sessions/owned"})) + }) + It("omits query tokens and response bodies from diagnostics", func() { + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusForbidden) + w.Write([]byte(`{"secret":"private-body"}`)) + })) + defer s.Close() + c := &Client{origin: s.URL, http: s.Client()} + _, err := c.Get(context.Background(), "/redfish/v1/Systems?$skiptoken=private-query") + Expect(Diagnostics(err)).To(Equal([]Diagnostic{{Code: "ACCESS_DENIED", Method: "GET", URI: "/redfish/v1/Systems", Status: 403, Stage: "observation"}})) + Expect(strings.Contains(err.Error(), "private")).To(BeFalse()) + Expect(CleanupReceipt(err)).To(BeEmpty()) + Expect(Diagnostics(nil)).To(BeEmpty()) + }) +}) diff --git a/pkg/dpf/redfish/evidence_test.go b/pkg/dpf/redfish/evidence_test.go new file mode 100644 index 00000000..8ff42de1 --- /dev/null +++ b/pkg/dpf/redfish/evidence_test.go @@ -0,0 +1,22 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package redfish + +import ( + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/gomega" +) + +// Success and partial/failure fixtures exercise the collector's shared return +// slot and every Redfish record that previously carried evidence references. +func expectNoRedfishEvidence(device inventory.Device, evidence map[string]inventory.Evidence) { + ExpectWithOffset(1, evidence).To(BeNil()) + ExpectWithOffset(1, device.EvidenceRefs).To(BeNil()) + for _, adapter := range device.Adapters { + ExpectWithOffset(1, adapter.EvidenceRefs).To(BeNil()) + } + for _, firmware := range device.Firmware { + ExpectWithOffset(1, firmware.EvidenceRefs).To(BeNil()) + } +} diff --git a/pkg/dpf/redfish/host_inventory.go b/pkg/dpf/redfish/host_inventory.go new file mode 100644 index 00000000..25340e21 --- /dev/null +++ b/pkg/dpf/redfish/host_inventory.go @@ -0,0 +1,276 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "fmt" + "reflect" + "slices" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +type hostCard struct { + resource Resource + owned bool + pci *Resource +} + +// collectHostAdapters joins only advertised resources of the selected System. +// Chassis membership alone is insufficient when a chassis is shared or its +// ownership is unknown; HPE's System PCI DeviceResourceId provides that join. +func collectHostAdapters(ctx context.Context, c *Client, system Resource, hostID string, hpe bool) ([]inventory.Adapter, error) { + cards := map[string]*hostCard{} + collections := map[string][]Resource{} + observed := map[string]Document{system.URI: hostIdentityFields(system.Document)} + readCollection := func(uri string) ([]Resource, error) { + members, err := c.Collection(ctx, uri) + if err == nil { + collections[uri] = members + for _, member := range members { + observed[member.URI] = hostIdentityFields(member.Document) + } + } + return members, err + } + putCard := func(resource Resource, owned bool) error { + key := strings.TrimSuffix(resource.URI, "/") + if existing := cards[key]; existing != nil { + if !reflect.DeepEqual(hostIdentityFields(existing.resource.Document), hostIdentityFields(resource.Document)) { + return &Error{Code: "INVENTORY_CHANGED", URI: resource.URI} + } + existing.owned = existing.owned || owned + } else { + cards[key] = &hostCard{resource: resource, owned: owned} + } + observed[resource.URI] = hostIdentityFields(resource.Document) + return nil + } + if uri := Link(system.Document, "NetworkInterfaces"); uri != "" { + interfaces, err := readCollection(uri) + if err != nil { + return nil, err + } + for _, iface := range interfaces { + uri := Link(Object(iface.Document, "Links"), "NetworkAdapter") + if uri == "" { + return nil, &Error{Code: "ADAPTER_LINK_MISSING", URI: iface.URI} + } + uri, err = c.guard(uri) + if err != nil { + return nil, err + } + card, err := c.Get(ctx, uri) + if err != nil { + return nil, err + } + if err := putCard(Resource{URI: uri, Document: card}, true); err != nil { + return nil, err + } + } + } + if hpe { + chassisLinks, err := c.resourceLinks(Object(system.Document, "Links"), "Chassis") + if err != nil { + return nil, err + } + for _, uri := range chassisLinks { + chassis, err := c.Get(ctx, uri) + if err != nil { + return nil, err + } + observed[uri] = hostIdentityFields(chassis) + owners, err := c.resourceLinks(Object(chassis, "Links"), "ComputerSystems") + if err != nil { + return nil, err + } + exclusive := len(owners) == 1 && strings.TrimSuffix(owners[0], "/") == strings.TrimSuffix(system.URI, "/") + networkURI := Link(chassis, "NetworkAdapters") + if networkURI == "" { + continue + } + adapters, err := readCollection(networkURI) + if err != nil { + return nil, err + } + for _, resource := range adapters { + if err := putCard(resource, exclusive); err != nil { + return nil, err + } + } + } + pciLinks, err := c.resourceLinks(Object(Object(Object(system.Document, "Oem"), "Hpe"), "Links"), "PCIDevices") + if err != nil { + return nil, err + } + for _, uri := range pciLinks { + devices, err := readCollection(uri) + if err != nil { + return nil, err + } + for _, pci := range devices { + class, hasClass := integer(pci.Document, "ClassCode", 255) + if (!hasClass || class != 2) && Text(pci.Document, "DeviceType") != "NIC" { + continue + } + id := Text(pci.Document, "DeviceResourceId") + var matches []*hostCard + if id != "" { + for _, card := range cards { + if Text(card.resource.Document, "Id") == id && card.pci == nil { + matches = append(matches, card) + } else if Text(card.resource.Document, "Id") == id && card.pci != nil { + return nil, &Error{Code: "ADAPTER_PCI_AMBIGUOUS", URI: pci.URI} + } + } + } + if len(matches) > 1 { + return nil, &Error{Code: "ADAPTER_PCI_AMBIGUOUS", URI: pci.URI} + } + if len(matches) == 1 { + matches[0].owned = true + matches[0].pci = &pci + } else { + // A PCI record is still useful when no NetworkAdapter exists. + cards[strings.TrimSuffix(pci.URI, "/")] = &hostCard{resource: pci, owned: true, pci: &pci} + } + } + } + } + if len(collections) == 0 { + return nil, &Error{Code: "HOST_INVENTORY_UNAVAILABLE", URI: system.URI} + } + var adapters []inventory.Adapter + serials := map[string]bool{} + for _, key := range sortedKeys(cards) { + card := cards[key] + if !card.owned { + continue + } + raw := Text(card.resource.Document, "SerialNumber") + serial := "" + state, reason := "unknown", "CARD_SERIAL_MISSING" + // Observation location survives replacement and newly available serial + // data. SerialNumber carries the separate physical identity. + adapterKey := inventory.Digest("dpf-adapter-observation-v1", []string{hostID, key}) + if ValidSerial(raw) { + serial = strings.TrimSpace(raw) + if serials[serial] { + return nil, &Error{Code: "CARD_IDENTITY_AMBIGUOUS", URI: card.resource.URI} + } + serials[serial] = true + state, reason = "known", "" + } + model := Text(card.resource.Document, "Model") + if model == "" { + model = Text(card.resource.Document, "Name") + } + a := inventory.Adapter{Key: adapterKey, SerialNumber: serial, SerialNumberRaw: raw, Model: model, IdentityState: state, ReasonCode: reason, Selection: "not-selected", Slot: Text(Object(Object(card.resource.Document, "Location"), "PartLocation"), "ServiceLabel")} + if card.pci != nil { + pci := card.pci.Document + a.PCIAddress = hpePCIAddress(pci) + a.VendorID = hexID(pci, "VendorID") + a.DeviceID = hexID(pci, "DeviceID") + if slot := Text(pci, "LocationString"); slot != "" { + a.Slot = slot + } + } + adapters = append(adapters, a) + } + // Recheck both collection membership and the identity/ownership fields used + // in joins. Telemetry counters are intentionally outside this snapshot. + for _, uri := range sortedKeys(collections) { + again, err := c.Collection(ctx, uri) + if err != nil { + return adapters, err + } + if !sameMembers(collections[uri], again) { + return adapters, &Error{Code: "INVENTORY_CHANGED", URI: uri} + } + } + for _, uri := range sortedKeys(observed) { + again, err := c.Get(ctx, uri) + if err != nil { + return adapters, err + } + if !reflect.DeepEqual(observed[uri], hostIdentityFields(again)) { + return adapters, &Error{Code: "IDENTITY_CHANGED", URI: uri} + } + } + return adapters, nil +} + +func sortedKeys[T any](m map[string]T) []string { + keys := make([]string, 0, len(m)) + for key := range m { + keys = append(keys, key) + } + slices.Sort(keys) + return keys +} + +// resourceLinks supports the array used by HPE OEM PCI links and standard links. +func (c *Client) resourceLinks(d Document, key string) ([]string, error) { + value, exists := d[key] + if !exists || value == nil { + return nil, nil + } + values, array := value.([]any) + if !array { + values = []any{value} + } + var links []string + seen := map[string]bool{} + for _, v := range values { + uri, err := c.guard(Link(Document{"v": v}, "v")) + if err != nil { + return nil, err + } + canonical := strings.TrimSuffix(uri, "/") + if seen[canonical] { + return nil, &Error{Code: "DUPLICATE_MEMBER", URI: canonical} + } + seen[canonical] = true + links = append(links, uri) + } + return links, nil +} + +func hostIdentityFields(d Document) Document { + result := Document{} + for _, key := range []string{"Id", "SerialNumber", "Model", "Name", "Links", "NetworkInterfaces", "NetworkAdapters", "Location", "DeviceResourceId", "ClassCode", "DeviceType", "SegmentNumber", "BusNumber", "DeviceNumber", "FunctionNumber", "VendorID", "DeviceID", "LocationString"} { + if value, ok := d[key]; ok { + result[key] = value + } + } + if pci := Object(Object(Object(d, "Oem"), "Hpe"), "Links")["PCIDevices"]; pci != nil { + result["HpePCIDevices"] = pci + } + return result +} +func integer(d Document, key string, max int) (int, bool) { + n, ok := d[key].(float64) + return int(n), ok && n >= 0 && n <= float64(max) && n == float64(int(n)) +} +func hpePCIAddress(d Document) string { + values := make([]int, 4) + for i, field := range []struct { + key string + max int + }{{"SegmentNumber", 65535}, {"BusNumber", 255}, {"DeviceNumber", 31}, {"FunctionNumber", 7}} { + value, ok := integer(d, field.key, field.max) + if !ok { + return "" + } + values[i] = value + } + return fmt.Sprintf("%04x:%02x:%02x.%x", values[0], values[1], values[2], values[3]) +} +func hexID(d Document, field string) string { + if value, ok := integer(d, field, 65535); ok { + return fmt.Sprintf("%04x", value) + } + return "" +} diff --git a/pkg/dpf/redfish/host_inventory_test.go b/pkg/dpf/redfish/host_inventory_test.go new file mode 100644 index 00000000..918af8d4 --- /dev/null +++ b/pkg/dpf/redfish/host_inventory_test.go @@ -0,0 +1,205 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func hostGraph() map[string]Document { + link := func(uri string) Document { return Document{"@odata.id": uri} } + collection := func(uris ...string) Document { + members := []any{} + for _, uri := range uris { + members = append(members, link(uri)) + } + return Document{"Members@odata.count": len(uris), "Members": members} + } + pci := func(id string, bus int, slot, name string) Document { + return Document{"ClassCode": 2, "DeviceType": "NIC", "DeviceResourceId": id, "SegmentNumber": 0, "BusNumber": bus, "DeviceNumber": 0, "FunctionNumber": 0, "VendorID": 5555, "DeviceID": 41695, "LocationString": slot, "Name": name} + } + return map[string]Document{ + "/redfish/v1/Systems/host": {"NetworkInterfaces": link("/redfish/v1/Systems/host/NetworkInterfaces"), "Links": Document{"Chassis": []any{link("/redfish/v1/Chassis/enclosure")}}, "Oem": Document{"Hpe": Document{"Links": Document{"PCIDevices": []any{link("/redfish/v1/Systems/host/PCIDevices")}}}}}, + "/redfish/v1/Systems/host/NetworkInterfaces": collection("/redfish/v1/Systems/host/NetworkInterfaces/embedded"), + "/redfish/v1/Systems/host/NetworkInterfaces/embedded": {"Links": Document{"NetworkAdapter": link("/redfish/v1/Chassis/enclosure/NetworkAdapters/embedded/")}}, + "/redfish/v1/Chassis/enclosure": {"NetworkAdapters": link("/redfish/v1/Chassis/enclosure/NetworkAdapters")}, + "/redfish/v1/Chassis/enclosure/NetworkAdapters": collection("/redfish/v1/Chassis/enclosure/NetworkAdapters/embedded", "/redfish/v1/Chassis/enclosure/NetworkAdapters/dpu"), + "/redfish/v1/Chassis/enclosure/NetworkAdapters/embedded": {"Id": "embedded", "Model": "Embedded NIC", "SerialNumber": "NIC-A"}, + "/redfish/v1/Chassis/enclosure/NetworkAdapters/dpu": {"Id": "dpu", "Model": "BlueField-4", "SerialNumber": "DPU-A"}, + "/redfish/v1/Systems/host/PCIDevices": collection("/redfish/v1/Systems/host/PCIDevices/dpu", "/redfish/v1/Systems/host/PCIDevices/other"), + "/redfish/v1/Systems/host/PCIDevices/dpu": pci("dpu", 129, "Slot 5", "BlueField-4"), + "/redfish/v1/Systems/host/PCIDevices/other": pci("", 33, "Slot 1", "ConnectX"), + } +} + +func graphClient(docs map[string]Document, mutate func(string, int, Document)) (*Client, func()) { + reads := map[string]int{} + s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + uri := strings.TrimSuffix(r.URL.Path, "/") + if r.Method != http.MethodGet { + w.WriteHeader(http.StatusMethodNotAllowed) + return + } + doc, ok := docs[uri] + if !ok { + w.WriteHeader(http.StatusNotFound) + return + } + reads[uri]++ + if mutate != nil { + mutate(uri, reads[uri], doc) + } + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(doc) + })) + return &Client{origin: s.URL, http: s.Client()}, s.Close +} + +var _ = Describe("host inventory ownership and partial identity", func() { + It("joins HPE chassis cards to System PCI records and retains cards without serials", func() { + c, closeServer := graphClient(hostGraph(), nil) + defer closeServer() + system, err := c.Get(context.Background(), "/redfish/v1/Systems/host") + Expect(err).NotTo(HaveOccurred()) + cards, err := collectHostAdapters(context.Background(), c, Resource{URI: "/redfish/v1/Systems/host", Document: system}, "host-a", true) + Expect(err).NotTo(HaveOccurred()) + Expect(cards).To(HaveLen(3)) + Expect(cards[0].SerialNumber).To(Equal("DPU-A")) + Expect(cards[0].PCIAddress).To(Equal("0000:81:00.0")) + Expect(cards[0].Slot).To(Equal("Slot 5")) + Expect(cards[0].VendorID).To(Equal("15b3")) + Expect(cards[0].DeviceID).To(Equal("a2df")) + Expect(cards[0].IdentityState).To(Equal("known")) + Expect(cards[0].EvidenceRefs).To(BeNil()) + Expect(cards[2].SerialNumber).To(BeEmpty()) + Expect(cards[2].ReasonCode).To(Equal("CARD_SERIAL_MISSING")) + Expect(cards[2].IdentityState).To(Equal("unknown")) + Expect(cards[2].PCIAddress).To(Equal("0000:21:00.0")) + Expect(cards[2].Key).NotTo(BeEmpty()) + }) + It("keeps the host resource observation key across replacement and newly exposed identity", func() { + keys := []string{} + for _, serial := range []string{"DPU-A", "DPU-REPLACED", ""} { + docs := hostGraph() + docs["/redfish/v1/Chassis/enclosure/NetworkAdapters/dpu"]["SerialNumber"] = serial + c, closeServer := graphClient(docs, nil) + system, err := c.Get(context.Background(), "/redfish/v1/Systems/host") + Expect(err).NotTo(HaveOccurred()) + cards, err := collectHostAdapters(context.Background(), c, Resource{URI: "/redfish/v1/Systems/host", Document: system}, "host-a", true) + closeServer() + Expect(err).NotTo(HaveOccurred()) + found := false + for _, card := range cards { + if card.Slot != "Slot 5" { + continue + } + found = true + Expect(card.SerialNumber).To(Equal(serial)) + keys = append(keys, card.Key) + } + Expect(found).To(BeTrue()) + } + Expect(keys).To(HaveLen(3)) + Expect(keys[1]).To(Equal(keys[0])) + Expect(keys[2]).To(Equal(keys[0])) + }) + It("keeps resource observation keys independent of collection order and bound to the host", func() { + keys := []string{} + for _, hostID := range []string{"host-a", "host-a", "host-b"} { + docs := hostGraph() + if len(keys) > 0 { + members := docs["/redfish/v1/Chassis/enclosure/NetworkAdapters"]["Members"].([]any) + members[0], members[1] = members[1], members[0] + } + c, closeServer := graphClient(docs, nil) + system, err := c.Get(context.Background(), "/redfish/v1/Systems/host") + Expect(err).NotTo(HaveOccurred()) + cards, err := collectHostAdapters(context.Background(), c, Resource{URI: "/redfish/v1/Systems/host", Document: system}, hostID, true) + closeServer() + Expect(err).NotTo(HaveOccurred()) + for _, card := range cards { + if card.SerialNumber == "DPU-A" { + keys = append(keys, card.Key) + } + } + } + Expect(keys).To(HaveLen(3)) + Expect(keys[1]).To(Equal(keys[0])) + Expect(keys[2]).NotTo(Equal(keys[0])) + }) + It("does not assign an uncorrelated shared-chassis adapter to the selected host", func() { + docs := hostGraph() + docs["/redfish/v1/Systems/host/PCIDevices/dpu"]["DeviceResourceId"] = "not-a-chassis-card" + c, closeServer := graphClient(docs, nil) + defer closeServer() + system, err := c.Get(context.Background(), "/redfish/v1/Systems/host") + Expect(err).NotTo(HaveOccurred()) + cards, err := collectHostAdapters(context.Background(), c, Resource{URI: "/redfish/v1/Systems/host", Document: system}, "host-a", true) + Expect(err).NotTo(HaveOccurred()) + for _, card := range cards { + Expect(card.SerialNumber).NotTo(Equal("DPU-A")) + } + }) + It("rejects duplicate physical serials and ambiguous PCI-to-adapter joins", func() { + for _, duplicateID := range []bool{false, true} { + docs := hostGraph() + if duplicateID { + docs["/redfish/v1/Chassis/enclosure/NetworkAdapters/embedded"]["Id"] = "dpu" + } else { + docs["/redfish/v1/Chassis/enclosure/NetworkAdapters/embedded"]["SerialNumber"] = "DPU-A" + } + c, closeServer := graphClient(docs, nil) + system, err := c.Get(context.Background(), "/redfish/v1/Systems/host") + Expect(err).NotTo(HaveOccurred()) + _, err = collectHostAdapters(context.Background(), c, Resource{URI: "/redfish/v1/Systems/host", Document: system}, "host-a", true) + Expect(err).To(HaveOccurred()) + if duplicateID { + Expect(Code(err)).To(Equal("ADAPTER_PCI_AMBIGUOUS")) + } else { + Expect(Code(err)).To(Equal("CARD_IDENTITY_AMBIGUOUS")) + } + closeServer() + } + }) + It("detects card replacement during collection", func() { + c, closeServer := graphClient(hostGraph(), func(uri string, reads int, doc Document) { + if uri == "/redfish/v1/Chassis/enclosure/NetworkAdapters/dpu" && reads > 1 { + doc["SerialNumber"] = "DPU-REPLACED" + } + }) + defer closeServer() + system, err := c.Get(context.Background(), "/redfish/v1/Systems/host") + Expect(err).NotTo(HaveOccurred()) + _, err = collectHostAdapters(context.Background(), c, Resource{URI: "/redfish/v1/Systems/host", Document: system}, "host-a", true) + Expect(Code(err)).To(Equal("IDENTITY_CHANGED")) + }) + It("still rechecks the owned System PCI join and vendor identity without reference metadata", func() { + for _, field := range []string{"DeviceResourceId", "VendorID"} { + c, closeServer := graphClient(hostGraph(), func(uri string, reads int, doc Document) { + if uri == "/redfish/v1/Systems/host/PCIDevices/dpu" && reads > 1 { + if field == "DeviceResourceId" { + doc[field] = "different-card" + } else { + doc[field] = 0x8086 + } + } + }) + system, err := c.Get(context.Background(), "/redfish/v1/Systems/host") + Expect(err).NotTo(HaveOccurred()) + _, err = collectHostAdapters(context.Background(), c, Resource{URI: "/redfish/v1/Systems/host", Document: system}, "host-a", true) + Expect(Code(err)).To(Equal("IDENTITY_CHANGED")) + closeServer() + } + }) + It("does not guess a PCI domain or address when components are unavailable", func() { + Expect(hpePCIAddress(Document{"BusNumber": float64(42)})).To(BeEmpty()) + Expect(hexID(Document{"VendorID": "15b3"}, "VendorID")).To(BeEmpty()) + }) +}) diff --git a/pkg/dpf/redfish/host_qualification_test.go b/pkg/dpf/redfish/host_qualification_test.go new file mode 100644 index 00000000..4c05f7f1 --- /dev/null +++ b/pkg/dpf/redfish/host_qualification_test.go @@ -0,0 +1,133 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("host BMC qualification metadata", func() { + It("reads manager firmware only for applicable catalog entries without collecting host software or time", func() { + for _, scenario := range []string{"no-catalog", "other-model", "no-verifier", "no-version-requirement", "qualified", "wrong-version", "denied", "ambiguous", "verification-failed"} { + docs := hostGraph() + docs["/redfish/v1/Systems/host"]["Manufacturer"] = "HPE" + docs["/redfish/v1/Systems/host"]["Model"] = "Host" + docs["/redfish/v1/Systems/host"]["UUID"] = "11111111-1111-4111-8111-111111111111" + docs["/redfish/v1/Systems/host"]["Actions"] = Document{"#ComputerSystem.Reset": Document{"ResetType@Redfish.AllowableValues": []any{"On", "ForceOff"}}} + link := func(uri string) Document { return Document{"@odata.id": uri} } + docs["/redfish/v1/Managers"] = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Managers/BMC")}} + docs["/redfish/v1/Managers/BMC"] = Document{"FirmwareVersion": "1.0", "DateTime": "2026-11-13T12:00:00Z", "NetworkProtocol": link("/redfish/v1/Managers/BMC/NetworkProtocol")} + if scenario == "ambiguous" { + docs["/redfish/v1/Managers"]["Members@odata.count"] = 2 + docs["/redfish/v1/Managers"]["Members"] = []any{link("/redfish/v1/Managers/BMC"), link("/redfish/v1/Managers/other")} + docs["/redfish/v1/Managers/other"] = Document{"FirmwareVersion": "1.0"} + } + reads := map[string]int{} + var mu sync.Mutex + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + uri := strings.TrimSuffix(r.URL.Path, "/") + mu.Lock() + reads[uri]++ + mu.Unlock() + var doc Document + switch uri { + case "/redfish/v1": + doc = Document{"Links": Document{"Sessions": link("/redfish/v1/SessionService/Sessions")}, "Systems": link("/redfish/v1/Systems"), "Managers": link("/redfish/v1/Managers"), "UpdateService": link("/redfish/v1/UpdateService")} + case "/redfish/v1/SessionService/Sessions": + w.Header().Set("X-Auth-Token", "fixture") + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/owned") + w.WriteHeader(http.StatusCreated) + doc = Document{} + case "/redfish/v1/SessionService/Sessions/owned": + w.WriteHeader(http.StatusNoContent) + return + case "/redfish/v1/Systems": + doc = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Systems/host")}} + default: + if scenario == "denied" && strings.HasPrefix(uri, "/redfish/v1/Managers") { + w.WriteHeader(http.StatusForbidden) + return + } + var ok bool + doc, ok = docs[uri] + if !ok { + w.WriteHeader(http.StatusNotFound) + return + } + } + _ = json.NewEncoder(w).Encode(doc) + })) + DeferCleanup(server.Close) + Expect(os.Setenv("DPF_HOST_QUALIFICATION_PASSWORD", "fixture")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_HOST_QUALIFICATION_PASSWORD") + source := fmt.Sprintf("credentials:\n read: {username: reader, passwordRef: {env: DPF_HOST_QUALIFICATION_PASSWORD}}\ndiscovery:\n tls: {insecureSkipVerify: true}\n hosts: [{bmcEndpoint: '%s', credentialRef: read}]\n probes: {mode: inventory}\n", server.URL) + in, err := config.Decode([]byte(source), filepath.Join(GinkgoT().TempDir(), "site.yaml")) + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(in) + Expect(err).NotTo(HaveOccurred()) + verified := 0 + qualification := Qualification{Adapter: "hpe-standard-v1", Model: "Host", BMCFirmware: "1.0", VerifyCompleteness: func(context.Context, Reader, Resource) error { + verified++ + if scenario == "verification-failed" { + return &Error{Code: "INCOMPLETE_COLLECTION"} + } + return nil + }} + switch scenario { + case "other-model": + qualification.Model = "Other" + case "no-verifier": + qualification.VerifyCompleteness = nil + case "no-version-requirement": + qualification.BMCFirmware = "" + case "wrong-version": + qualification.BMCFirmware = "2.0" + } + collector := Collector{Qualifications: []Qualification{qualification}} + if scenario == "no-catalog" { + collector.Qualifications = nil + } + device, _, err := collector.Collect(context.Background(), req.Targets[0], time.Second) + Expect(err).NotTo(HaveOccurred(), scenario) + Expect(device.Firmware).To(BeEmpty(), scenario) + Expect(device.Time).To(BeZero(), scenario) + Expect(device.Observations).To(BeEmpty(), scenario) + Expect(device.AdvertisedResetTypes).To(Equal([]string{"On", "ForceOff"}), scenario) + Expect(device.Adapters).To(ContainElement(And(HaveField("SerialNumber", "DPU-A"), HaveField("PCIAddress", "0000:81:00.0"))), scenario) + if scenario == "qualified" { + Expect(device.Qualification).To(Equal("qualified")) + Expect(device.InventoryCompleteness).To(Equal("complete")) + } else { + Expect(device.Qualification).To(Equal("unqualified"), scenario) + } + if scenario == "qualified" || scenario == "verification-failed" { + Expect(verified).To(Equal(1), scenario) + } else { + Expect(verified).To(BeZero(), scenario) + } + mu.Lock() + Expect(reads["/redfish/v1/UpdateService"]).To(BeZero(), scenario) + Expect(reads["/redfish/v1/Managers/BMC/NetworkProtocol"]).To(BeZero(), scenario) + Expect(reads["/redfish/v1/Systems/host"]).To(BeNumerically(">=", 3), scenario) + if scenario == "no-catalog" || scenario == "other-model" || scenario == "no-verifier" || scenario == "no-version-requirement" { + Expect(reads["/redfish/v1/Managers"]).To(BeZero(), scenario) + } else { + Expect(reads["/redfish/v1/Managers"]).To(BeNumerically(">", 0), scenario) + } + mu.Unlock() + } + }) +}) diff --git a/pkg/dpf/redfish/inventory.go b/pkg/dpf/redfish/inventory.go new file mode 100644 index 00000000..4a5bf371 --- /dev/null +++ b/pkg/dpf/redfish/inventory.go @@ -0,0 +1,343 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "reflect" + "regexp" + "slices" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +// Qualification is supplied by the installed, reviewed hardware catalog, never site YAML. +// An empty catalog collects candidate facts while keeping completeness unqualified. +type Qualification struct { + VerifyCompleteness func(context.Context, Reader, Resource) error + Adapter string + Model string + BMCFirmware string +} +type Collector struct{ Qualifications []Qualification } + +func ValidSerial(raw string) bool { + s := strings.TrimSpace(raw) + switch strings.ToLower(s) { + case "", "unknown", "n/a", "none", "to be filled by o.e.m.", "default string": + return false + } + return strings.Trim(s, "0- ") != "" +} + +// Collect returns typed Redfish facts without evidence metadata; the shared map return stays nil. +func (co Collector) Collect(ctx context.Context, target config.Target, timeout time.Duration) (device inventory.Device, evidence map[string]inventory.Evidence, err error) { + started := time.Now() + label := target.DisplayName() + logger := log.FromContext(ctx).WithName("redfish").WithValues("endpoint", label, "role", target.Role) + ctx = log.IntoContext(ctx, logger) + output := ui.FromContext(ctx) + stage := "" + begin := func(name string) { + stage = name + output.Info("[RUNNING] %s: %s", label, stage) + logger.V(1).Info("Redfish stage started", "stage", stage) + } + pass := func(detail string) { + output.Info("[PASS] %s: %s — %s", label, stage, detail) + stage = "" + } + device = inventory.Device{RecordKey: inventory.Digest("dpf-access-v1", []string{target.Role, target.Endpoint, target.SystemURI}), InputRef: target.InputRef, BMCEndpoint: target.Endpoint, ObservationState: "unavailable", Qualification: "unqualified", InventoryCompleteness: "unknown", Authentication: inventory.Authentication{PasswordRead: "unknown", ClientCertificateSupport: "unknown", DPFClientAuthentication: "not-evaluated"}} + if target.Role == "dpus" { + device.Time = inventory.TimeObservation{State: "unknown", ReasonCode: "PROPERTY_NOT_EXPOSED"} + } + expectedSerials := map[string]string{} + bf4Identities := map[string]Document{} + var psidCard Resource + var currentPSIDCard Document + c := New(target, timeout) + defer func() { + if stage != "" { + output.Info("[FAIL] %s: %s (%s)", label, stage, Code(err)) + } + cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 15*time.Second) + defer cancel() + if cleanupErr := c.Close(cleanupCtx); cleanupErr != nil { + err = &ObservationError{Primary: err, Cleanup: cleanupErr, SessionURI: c.session} + device.ReasonCode = "CLEANUP_INCOMPLETE" + } + if err != nil && psidCard.URI != "" { + state := device.Observations["psid"] + if state.State == "observed" || state.ReasonCode == "PSID_NOT_VERIFIED" { + device.Observations["psid"] = psidUnavailable("PSID_RECHECK_FAILED", "A required identity, source or cleanup recheck failed; the Redfish PSID is ineligible.") + } + } + if err != nil && device.ReasonCode == "" { + device.ReasonCode = Code(err) + } + logger.V(1).Info("Redfish collection completed", "identityState", device.ObservationState, "identityEstablished", device.ID != "", "reasonCode", device.ReasonCode, "duration", time.Since(started)) + }() + begin("Redfish connection/session") + root, err := c.Open(ctx) + if err != nil { + device.ReasonCode = Code(err) + if target.Role == "dpus" { + state := unavailable(err) + if Code(err) == "PASSWORD_CHANGE_REQUIRED" { + state.Message = "The BMC requires a password change; discovery does not change passwords." + } + device.Observations = map[string]inventory.Observation{"bmc-administrator": state} + } + return device, evidence, err + } + pass("session established") + device.Authentication.Method = "session" + + if target.Role == "dpus" { + begin("DPU BMC administrator privileges") + admin := c.administrator(ctx, root) + device.Observations = map[string]inventory.Observation{"bmc-administrator": admin} + logger.V(1).Info("BMC administrator verification completed", "state", admin.State, "reasonCode", admin.ReasonCode) + if admin.State == "observed" && admin.ReasonCode == "BMC_ADMIN_VERIFIED" { + pass("verified") + } else { + output.Info("[FAIL] %s: %s (%s)", label, stage, admin.ReasonCode) + stage = "" + } + } + begin("Physical inventory") + systems, err := c.Collection(ctx, Link(root, "Systems")) + if err != nil { + return device, evidence, err + } + device.Authentication.PasswordRead = "verified" + var system Resource + if target.SystemURI != "" { + for _, s := range systems { + if s.URI == target.SystemURI { + system = s + } + } + } else if len(systems) == 1 { + system = systems[0] + } + if system.URI == "" { + device.ObservationState = "ambiguous" + return device, evidence, &Error{Code: "SYSTEM_SELECTION_REQUIRED"} + } + device.SystemURI = system.URI + device.Manufacturer = Text(system.Document, "Manufacturer") + device.Model = Text(system.Document, "Model") + device.SystemUUID = Text(system.Document, "UUID") + bf4 := target.Role == "dpus" && b4240Service(root, nil) + if target.Role == "dpus" { + device.Observations["psid"] = inventory.Observation{State: "unknown", ReasonCode: "PSID_READER_UNSUPPORTED", Message: "This device is outside the supported BF4 AssetTag PSID reader.", Tool: "redfish"} + } + if bf4 && !b4240Service(root, system.Document) { + return device, evidence, &Error{Code: "DPU_IDENTITY_CONFLICT"} + } + adapter := "" + maker := strings.ToLower(device.Manufacturer) + switch { + case bf4: + adapter = "bluefield-card-v1" + case strings.Contains(maker, "nvidia") || strings.Contains(maker, "mellanox"): + adapter = "bluefield-card-v1" + case strings.Contains(maker, "hpe") || strings.Contains(maker, "hewlett"): + adapter = "hpe-standard-v1" + case strings.Contains(maker, "dell"): + adapter = "dell-standard-v1" + } + device.ReaderAdapter = adapter + action := Object(Object(system.Document, "Actions"), "#ComputerSystem.Reset") + if values, ok := action["ResetType@Redfish.AllowableValues"].([]any); ok { + for _, v := range values { + if value, ok := v.(string); ok { + device.AdvertisedResetTypes = append(device.AdvertisedResetTypes, value) + } + } + } + if target.Role == "dpus" { + if adapter != "bluefield-card-v1" { + return device, evidence, &Error{Code: "UNSUPPORTED_DPU_IDENTITY"} + } + chassis, er := c.Collection(ctx, Link(root, "Chassis")) + if er != nil { + return device, evidence, er + } + var cards []Resource + if bf4 { + var witnesses []Resource + cards, witnesses = c.b4240Cards(system, chassis) + for _, resources := range [][]Resource{cards, witnesses} { + for _, resource := range resources { + bf4Identities[resource.URI] = cardIdentity(resource.Document) + } + } + } else { + for _, ch := range chassis { + model := Text(ch.Document, "Model") + if strings.Contains(strings.ToLower(model), "bluefield") && ValidSerial(Text(ch.Document, "SerialNumber")) && c.linkedToSystem(ch.Document, system.URI) { + cards = append(cards, ch) + } + } + } + if len(cards) != 1 { + return device, evidence, &Error{Code: "CARD_IDENTITY_AMBIGUOUS"} + } + if bf4 { + psidCard = cards[0] + device.Observations["psid"] = inventory.Observation{State: "unknown", ReasonCode: "PSID_NOT_VERIFIED", Message: "Redfish PSID eligibility awaits successful identity and source rechecks.", Tool: "redfish"} + } + device.SerialNumberRaw = Text(cards[0].Document, "SerialNumber") + if !bf4 { + expectedSerials[cards[0].URI] = device.SerialNumberRaw + } + device.SerialNumber = strings.TrimSpace(device.SerialNumberRaw) + device.Model = Text(cards[0].Document, "Model") + device.IdentityKey = []string{"nvidia", device.SerialNumber} + device.ID = inventory.PhysicalID("dpu", device.IdentityKey) + } else { + if adapter != "hpe-standard-v1" && adapter != "dell-standard-v1" { + return device, evidence, &Error{Code: "UNSUPPORTED_HOST_ADAPTER"} + } + if !validHostUUID(device.SystemUUID) { + return device, evidence, &Error{Code: "HOST_IDENTITY_MISSING"} + } + device.IdentityKey = []string{"system-uuid", strings.ToLower(device.SystemUUID)} + device.ID = inventory.PhysicalID("host", device.IdentityKey) + device.Adapters, err = collectHostAdapters(ctx, c, system, device.ID, adapter == "hpe-standard-v1") + if err != nil { + return device, evidence, err + } + device.InventoryCompleteness = "unqualified" + } + output.Info("[RUNNING] %s: Physical inventory read completed; identity recheck pending", label) + stage = "" + bmcFirmware := "" + if target.Role == "dpus" { + begin("Optional BMC firmware/time/settings observations") + bmcFirmware = collectSoftware(ctx, c, root, system, &device) + observed, unavailableCount := 0, 0 + for _, key := range []string{"managers", "firmware-inventory", "bmc-time", "bmc-ntp"} { + if device.Observations[key].State == "observed" { + observed++ + } else { + unavailableCount++ + } + } + for _, firmware := range device.Firmware { + if firmware.State == "known" { + observed++ + } else { + unavailableCount++ + } + } + if unavailableCount > 0 { + output.Info("[UNAVAILABLE] %s: Optional BMC firmware/time/settings reads completed; %d observed, %d unavailable or unknown; inspect the detailed report for reasons, compatibility not evaluated", label, observed, unavailableCount) + stage = "" + } else { + pass("reads completed; compatibility not evaluated") + } + logger.V(1).Info("Optional Redfish observations completed", "observedSections", observed, "unavailableObservations", unavailableCount) + } else { + bmcFirmware = co.hostQualificationFirmware(ctx, c, root, adapter, device.Model) + } + for _, q := range co.Qualifications { + if q.Adapter == adapter && q.Model == device.Model && q.BMCFirmware == bmcFirmware && bmcFirmware != "" && q.VerifyCompleteness != nil && q.VerifyCompleteness(ctx, c, system) == nil { + device.Qualification = "qualified" + device.InventoryCompleteness = "complete" + } + } + begin("Physical identity recheck") + for uri, identity := range bf4Identities { + current, e := c.Get(ctx, uri) + if e != nil { + return device, evidence, e + } + if !reflect.DeepEqual(cardIdentity(current), identity) { + return device, evidence, &Error{Code: "IDENTITY_CHANGED", URI: uri} + } + if uri == psidCard.URI { + currentPSIDCard = current + } + } + for uri, raw := range expectedSerials { + current, e := c.Get(ctx, uri) + if e != nil { + return device, evidence, e + } + if Text(current, "SerialNumber") != raw { + return device, evidence, &Error{Code: "IDENTITY_CHANGED", URI: uri} + } + } + refreshed, er := c.Get(ctx, system.URI) + if er != nil { + return device, evidence, er + } + if Text(refreshed, "UUID") != device.SystemUUID || Text(refreshed, "SerialNumber") != Text(system.Document, "SerialNumber") || len(bf4Identities) > 0 && !reflect.DeepEqual(cardIdentity(refreshed), cardIdentity(system.Document)) { + return device, evidence, &Error{Code: "IDENTITY_CHANGED", URI: system.URI} + } + if bf4 { + device.Observations["psid"] = bf4PSID(system.Document, psidCard.Document, refreshed, currentPSIDCard) + } + if target.Role == "dpus" && c.passwordChangeRequired { + device.Observations["bmc-administrator"] = inventory.Observation{State: "failed", ReasonCode: "BMC_PASSWORD_CHANGE_REQUIRED", Message: "The BMC requires a password change; discovery does not change passwords.", Tool: "redfish"} + output.Info("[FAIL] %s: DPU BMC administrator privileges (BMC_PASSWORD_CHANGE_REQUIRED)", label) + } + device.ObservationState = "observed" + pass("verified; session cleanup pending") + return device, evidence, nil +} + +// hostQualificationFirmware reads only manager metadata when an applicable catalog +// entry needs it. Host firmware inventory and clock settings are outside discovery. +func (co Collector) hostQualificationFirmware(ctx context.Context, reader Reader, root Document, adapter, model string) string { + for _, q := range co.Qualifications { + if q.Adapter != adapter || q.Model != model || q.BMCFirmware == "" || q.VerifyCompleteness == nil { + continue + } + manager, state := softwareManager(ctx, reader, root) + if state.State == "observed" { + return Text(manager.Document, "FirmwareVersion") + } + return "" + } + return "" +} + +func sameMembers(a, b []Resource) bool { + keys := func(r []Resource) []string { + s := make([]string, 0, len(r)) + for _, v := range r { + s = append(s, v.URI) + } + slices.Sort(s) + return s + } + return reflect.DeepEqual(keys(a), keys(b)) +} + +func (c *Client) linkedToSystem(card Document, system string) bool { + links := Object(card, "Links") + if values, ok := links["ComputerSystems"].([]any); ok { + for _, value := range values { + uri, err := c.guard(Link(Document{"v": value}, "v")) + if err == nil && uri == system { + return true + } + } + } + return false +} + +var hostUUIDPattern = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`) + +func validHostUUID(s string) bool { + return hostUUIDPattern.MatchString(s) && strings.Trim(s, "0-") != "" && strings.Trim(strings.ToLower(s), "f-") != "" +} diff --git a/pkg/dpf/redfish/inventory_test.go b/pkg/dpf/redfish/inventory_test.go new file mode 100644 index 00000000..59e93102 --- /dev/null +++ b/pkg/dpf/redfish/inventory_test.go @@ -0,0 +1,130 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "encoding/json" + "encoding/pem" + "fmt" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "time" +) + +var _ = Describe("physical card identity", func() { + It("records the linked card and rejects a replacement at the same URI", func() { + for _, scenario := range []struct{ replace, absolute bool }{{false, false}, {true, false}, {false, true}, {true, true}} { + replace := scenario.replace + cardReads := 0 + writes := []string{} + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != "GET" { + writes = append(writes, r.Method+" "+r.URL.Path) + } + w.Header().Set("Content-Type", "application/json") + var doc any + switch r.URL.Path { + case "/redfish/v1/": + doc = Document{"SessionService": Document{"@odata.id": "/redfish/v1/SessionService"}, "Systems": Document{"@odata.id": "/redfish/v1/Systems"}, "Chassis": Document{"@odata.id": "/redfish/v1/Chassis"}} + case "/redfish/v1/SessionService": + doc = Document{"Sessions": Document{"@odata.id": "/redfish/v1/SessionService/Sessions"}} + case "/redfish/v1/SessionService/Sessions": + w.Header().Set("X-Auth-Token", "private-session") + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/owned") + w.WriteHeader(201) + doc = Document{} + case "/redfish/v1/SessionService/Sessions/owned": + w.WriteHeader(204) + return + case "/redfish/v1/Systems": + doc = Document{"Members@odata.count": 1, "Members": []any{Document{"@odata.id": "/redfish/v1/Systems/selected"}}} + case "/redfish/v1/Systems/selected": + doc = Document{"UUID": "11111111-1111-4111-8111-111111111111", "Manufacturer": "NVIDIA", "Model": "BlueField-3"} + case "/redfish/v1/Chassis": + doc = Document{"Members@odata.count": 1, "Members": []any{Document{"@odata.id": "/redfish/v1/Chassis/Card"}}} + case "/redfish/v1/Chassis/Card": + cardReads++ + serial := "CARD-A" + if replace && cardReads > 1 { + serial = "CARD-B" + } + doc = Document{"Model": "BlueField-3", "SerialNumber": serial, "Links": Document{"ComputerSystems": []any{Document{"@odata.id": "/redfish/v1/Systems/selected"}}}} + default: + w.WriteHeader(404) + return + } + if scenario.absolute { + absoluteLinks(doc, "https://"+r.Host) + } + json.NewEncoder(w).Encode(doc) + })) + dir, e := os.MkdirTemp("", "dpf-card-") + Expect(e).NotTo(HaveOccurred()) + ca := filepath.Join(dir, "ca.pem") + Expect(os.WriteFile(ca, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw}), 0600)).To(Succeed()) + Expect(os.Setenv("DPF_CARD_PASSWORD", "private-password")).To(Succeed()) + source := fmt.Sprintf("management:\n kubeconfigRef: {file: unused}\ncredentials:\n read: {username: reader, passwordRef: {env: DPF_CARD_PASSWORD}}\ndiscovery:\n tls: {caFile: '%s'}\n dpus: [{bmcEndpoint: '%s', credentialRef: read, systemURI: /redfish/v1/Systems/selected}]\n hosts: [{bmcEndpoint: 'https://192.0.2.1', credentialRef: read}]\n", ca, server.URL) + in, e := config.Decode([]byte(source), filepath.Join(dir, "site.yaml")) + Expect(e).NotTo(HaveOccurred()) + req, e := config.Resolve(in) + Expect(e).NotTo(HaveOccurred()) + device, evidence, e := (Collector{}).Collect(context.Background(), req.Targets[0], time.Second) + expectNoRedfishEvidence(device, evidence) + if replace { + Expect(e).To(HaveOccurred()) + Expect(Code(e)).To(Equal("IDENTITY_CHANGED")) + } else { + Expect(e).NotTo(HaveOccurred()) + Expect(device.SerialNumber).To(Equal("CARD-A")) + Expect(device.ObservationState).To(Equal("observed")) + Expect(device.Qualification).To(Equal("unqualified")) + Expect(device.Observations["psid"].ReasonCode).To(Equal("PSID_READER_UNSUPPORTED")) + } + Expect(writes).To(Equal([]string{"POST /redfish/v1/SessionService/Sessions", "DELETE /redfish/v1/SessionService/Sessions/owned"})) + server.Close() + os.RemoveAll(dir) + os.Unsetenv("DPF_CARD_PASSWORD") + } + }) + It("uses ComputerSystem linkage rather than the first chassis", func() { + card := Document{"Links": Document{"ComputerSystems": []any{Document{"@odata.id": "/redfish/v1/Systems/selected"}}}} + Expect((&Client{origin: "https://bmc.invalid"}).linkedToSystem(card, "/redfish/v1/Systems/selected")).To(BeTrue()) + Expect((&Client{origin: "https://bmc.invalid"}).linkedToSystem(card, "/redfish/v1/Systems/other")).To(BeFalse()) + card["Links"] = Document{"ComputerSystems": []any{Document{"@odata.id": "https://foreign.invalid/redfish/v1/Systems/selected"}}} + Expect((&Client{origin: "https://bmc.invalid"}).linkedToSystem(card, "/redfish/v1/Systems/selected")).To(BeFalse()) + }) +}) + +var _ = Describe("host UUID syntax", func() { + It("rejects placeholders before assigning identity", func() { + for _, value := range []string{"unknown", "default", "00000000-0000-0000-0000-000000000000", "ffffffff-ffff-ffff-ffff-ffffffffffff", "bad-uuid"} { + Expect(validHostUUID(value)).To(BeFalse()) + } + Expect(validHostUUID("11111111-1111-4111-8111-111111111111")).To(BeTrue()) + }) +}) + +func absoluteLinks(value any, origin string) { + switch v := value.(type) { + case Document: + for key, item := range v { + if key == "@odata.id" { + if s, ok := item.(string); ok { + v[key] = origin + s + } + } else { + absoluteLinks(item, origin) + } + } + case []any: + for _, item := range v { + absoluteLinks(item, origin) + } + } +} diff --git a/pkg/dpf/redfish/progress_test.go b/pkg/dpf/redfish/progress_test.go new file mode 100644 index 00000000..86c45e2f --- /dev/null +++ b/pkg/dpf/redfish/progress_test.go @@ -0,0 +1,222 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package redfish + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/go-logr/zapr" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "go.uber.org/zap" + "go.uber.org/zap/zapcore" + "go.uber.org/zap/zaptest/observer" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +type progressBuffer struct { + mu sync.Mutex + b bytes.Buffer +} + +func (b *progressBuffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.b.Write(p) +} +func (b *progressBuffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.b.String() +} + +var _ = Describe("secret-safe Redfish progress and logs", func() { + It("selects debug/trace metadata without logging URLs, bodies, tokens or errors", func() { + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write([]byte(`{"private":"response-body-canary"}`)) + })) + DeferCleanup(server.Close) + for _, level := range []zapcore.Level{zapcore.InfoLevel, zapcore.DebugLevel, -2} { + core, records := observer.New(level) + ctx := log.IntoContext(context.Background(), zapr.NewLogger(zap.New(core))) + c := &Client{origin: server.URL, http: server.Client(), token: "token-canary", session: "/redfish/v1/SessionService/Sessions/session-id-canary"} + _, err := c.Get(ctx, "/redfish/v1/AccountService/Accounts/username-canary?$skiptoken=query-canary") + Expect(Code(err)).To(Equal("ACCESS_DENIED")) + _, err = c.Get(ctx, "https://userinfo-canary:password-canary@foreign.invalid/redfish/v1/Systems") + Expect(err).To(HaveOccurred()) + Expect(Code(c.Close(ctx))).To(Equal("ACCESS_DENIED")) + if level == zapcore.InfoLevel { + Expect(records.Len()).To(BeZero()) + } else { + Expect(records.FilterMessage("Redfish session cleanup completed").Len()).To(Equal(1)) + } + if level == -2 { + requests := records.FilterMessage("Redfish request completed").All() + Expect(requests).To(HaveLen(3)) + Expect(requests[0].ContextMap()).To(HaveKeyWithValue("httpStatus", int64(403))) + Expect(requests[0].ContextMap()).To(HaveKey("duration")) + } else { + Expect(records.FilterMessage("Redfish request completed").Len()).To(BeZero()) + } + encoded, err := json.Marshal(records.All()) + Expect(err).NotTo(HaveOccurred()) + for _, secret := range []string{"response-body-canary", "token-canary", "session-id-canary", "username-canary", "query-canary", "userinfo-canary", "password-canary"} { + Expect(string(encoded)).NotTo(ContainSubstring(secret)) + } + } + }) + + It("emits live stages without host evidence on success or deferred cleanup failure", func() { + for _, cleanupFails := range []bool{false, true} { + docs := hostGraph() + docs["/redfish/v1/Systems/host"]["Manufacturer"] = "HPE" + docs["/redfish/v1/Systems/host"]["Model"] = "Host" + docs["/redfish/v1/Systems/host"]["UUID"] = "11111111-1111-4111-8111-111111111111" + link := func(uri string) Document { return Document{"@odata.id": uri} } + if !cleanupFails { + docs["/redfish/v1/Managers"] = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Managers/BMC")}} + docs["/redfish/v1/Managers/BMC"] = Document{"FirmwareVersion": "1.0", "DateTime": "2026-11-13T12:00:00Z", "NetworkProtocol": link("/redfish/v1/Managers/BMC/NetworkProtocol")} + docs["/redfish/v1/Managers/BMC/NetworkProtocol"] = Document{"NTP": Document{"ProtocolEnabled": true, "NTPServers": []any{"ntp.example.invalid"}}} + docs["/redfish/v1/UpdateService"] = Document{"FirmwareInventory": link("/redfish/v1/UpdateService/FirmwareInventory")} + members := []any{} + for _, component := range []string{"bmc", "cec", "atf", "uefi", "nic", "dpu-os", "dpu-bsp", "golden"} { + uri := "/redfish/v1/UpdateService/FirmwareInventory/" + component + members = append(members, link(uri)) + docs[uri] = Document{"Id": component, "Version": "1.0"} + } + docs["/redfish/v1/UpdateService/FirmwareInventory"] = Document{"Members@odata.count": len(members), "Members": members} + } + entered, rechecking, cleaning := make(chan struct{}), make(chan struct{}), make(chan struct{}) + releaseRead, releaseRecheck, releaseCleanup := make(chan struct{}), make(chan struct{}), make(chan struct{}) + release := func(ch chan struct{}) { + select { + case <-ch: + default: + close(ch) + } + } + var systemReads, softwareReads atomic.Int32 + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + uri := strings.TrimSuffix(r.URL.Path, "/") + if strings.HasPrefix(uri, "/redfish/v1/Managers") || strings.HasPrefix(uri, "/redfish/v1/UpdateService") { + softwareReads.Add(1) + } + var doc Document + switch uri { + case "/redfish/v1": + doc = Document{"Links": Document{"Sessions": link("/redfish/v1/SessionService/Sessions")}, "Systems": link("/redfish/v1/Systems"), "private": "response-body-canary"} + if !cleanupFails { + doc["Managers"] = link("/redfish/v1/Managers") + doc["UpdateService"] = link("/redfish/v1/UpdateService") + } + case "/redfish/v1/SessionService/Sessions": + w.Header().Set("X-Auth-Token", "token-canary") + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/session-id-canary") + w.WriteHeader(http.StatusCreated) + doc = Document{} + case "/redfish/v1/SessionService/Sessions/session-id-canary": + close(cleaning) + <-releaseCleanup + if cleanupFails { + w.WriteHeader(http.StatusForbidden) + } else { + w.WriteHeader(http.StatusNoContent) + } + return + case "/redfish/v1/Systems": + close(entered) + <-releaseRead + doc = Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Systems/host")}} + default: + var ok bool + doc, ok = docs[uri] + if !ok { + w.WriteHeader(http.StatusNotFound) + return + } + if uri == "/redfish/v1/Systems/host" && systemReads.Add(1) == 3 { + close(rechecking) + <-releaseRecheck + } + } + _ = json.NewEncoder(w).Encode(doc) + })) + DeferCleanup(func() { release(releaseRead); release(releaseRecheck); release(releaseCleanup); server.Close() }) + Expect(os.Setenv("DPF_PROGRESS_PASSWORD", "password-canary")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_PROGRESS_PASSWORD") + source := fmt.Sprintf("credentials:\n read: {username: username-canary, passwordRef: {env: DPF_PROGRESS_PASSWORD}}\ndiscovery:\n tls: {insecureSkipVerify: true}\n hosts: [{name: 'Named host', bmcEndpoint: '%s', credentialRef: read}]\n probes: {mode: inventory}\n", server.URL) + in, err := config.Decode([]byte(source), filepath.Join(GinkgoT().TempDir(), "site.yaml")) + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(in) + Expect(err).NotTo(HaveOccurred()) + core, records := observer.New(zapcore.Level(-2)) + ctx, cancel := context.WithCancel(log.IntoContext(context.Background(), zapr.NewLogger(zap.New(core)))) + DeferCleanup(cancel) + progress := &progressBuffer{} + ctx = ui.WithOutput(ctx, ui.NewWithWriter(progress)) + type collected struct { + device inventory.Device + evidence map[string]inventory.Evidence + err error + } + result := make(chan collected, 1) + go func() { + device, evidence, err := (Collector{}).Collect(ctx, req.Targets[0], 3*time.Second) + result <- collected{device: device, evidence: evidence, err: err} + }() + Eventually(entered, 3*time.Second).Should(BeClosed()) + Expect(progress.String()).To(ContainSubstring("[RUNNING] \"Named host\"")) + Expect(progress.String()).To(ContainSubstring(": Physical inventory")) + Expect(progress.String()).NotTo(ContainSubstring("[PASS] \"Named host\" (" + server.URL + "): Physical")) + release(releaseRead) + Eventually(rechecking, 3*time.Second).Should(BeClosed()) + Expect(progress.String()).NotTo(ContainSubstring("[UNAVAILABLE]")) + Expect(progress.String()).NotTo(ContainSubstring("Optional BMC firmware/time/settings")) + Expect(progress.String()).NotTo(ContainSubstring(": Physical identity recheck — verified")) + release(releaseRecheck) + Eventually(cleaning, 3*time.Second).Should(BeClosed()) + Expect(records.FilterMessage("Redfish collection completed").Len()).To(BeZero()) + Expect(progress.String()).NotTo(ContainSubstring(": Redfish session cleanup completed")) + cancel() // Deferred cleanup has its own context and must still finish. + release(releaseCleanup) + var collection collected + Eventually(result, 3*time.Second).Should(Receive(&collection)) + expectNoRedfishEvidence(collection.device, collection.evidence) + Expect(softwareReads.Load()).To(BeZero()) + Expect(collection.device.Firmware).To(BeEmpty()) + Expect(collection.device.Time).To(BeZero()) + Expect(collection.device.Observations).To(BeEmpty()) + Expect(collection.device.Adapters).To(ContainElement(And(HaveField("SerialNumber", "DPU-A"), HaveField("PCIAddress", "0000:81:00.0")))) + collectionErr := collection.err + if cleanupFails { + Expect(collectionErr).To(HaveOccurred()) + Expect(progress.String()).To(ContainSubstring(": Redfish session cleanup (ACCESS_DENIED)")) + } else { + Expect(collectionErr).NotTo(HaveOccurred()) + Expect(progress.String()).To(ContainSubstring(": Redfish session cleanup completed")) + } + Expect(records.FilterMessage("Redfish collection completed").Len()).To(Equal(1)) + encoded, err := json.Marshal(records.All()) + Expect(err).NotTo(HaveOccurred()) + for _, secret := range []string{"password-canary", "username-canary", "token-canary", "session-id-canary", "response-body-canary"} { + Expect(progress.String() + string(encoded)).NotTo(ContainSubstring(secret)) + } + } + }) +}) diff --git a/pkg/dpf/redfish/psid.go b/pkg/dpf/redfish/psid.go new file mode 100644 index 00000000..a4e09695 --- /dev/null +++ b/pkg/dpf/redfish/psid.go @@ -0,0 +1,59 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package redfish + +import ( + "reflect" + "regexp" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +var psidToken = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$`) + +func psidUnavailable(code, message string) inventory.Observation { + return inventory.Observation{State: "unavailable", ReasonCode: code, Message: message, Tool: "redfish"} +} + +// bf4PSID uses the supported BF4 reader's AssetTag semantics: System first, +// then the identity-selected chassis. Both advertised values must agree and +// remain unchanged through the existing identity rereads. No raw documents or +// malformed AssetTag values are persisted, and no SSH runtime is manufactured. +func bf4PSID(system, card, currentSystem, currentCard Document) inventory.Observation { + for _, source := range []struct{ initial, current Document }{{system, currentSystem}, {card, currentCard}} { + initial, initialPresent := source.initial["AssetTag"] + current, currentPresent := source.current["AssetTag"] + if initialPresent != currentPresent || !reflect.DeepEqual(initial, current) { + return psidUnavailable("PSID_CHANGED", "A Redfish PSID source changed during collection; rerun discovery before using this PSID.") + } + } + value := "" + for _, source := range []Document{system, card} { + raw, present := source["AssetTag"] + if !present || raw == nil { + continue + } + text, ok := raw.(string) + if !ok { + return psidUnavailable("PSID_INVALID", "A supported BF4 AssetTag is not a valid PSID identifier string.") + } + if strings.TrimSpace(text) == "" || strings.EqualFold(strings.TrimSpace(text), "N/A") { + continue + } + if !psidToken.MatchString(text) { + return psidUnavailable("PSID_INVALID", "A supported BF4 AssetTag is not a valid bounded ASCII PSID identifier.") + } + if value != "" && value != text { + return psidUnavailable("PSID_CONFLICT", "The supported BF4 System and chassis AssetTags report contradictory PSIDs.") + } + if value == "" { + value = text + } + } + if value == "" { + return inventory.Observation{State: "unknown", ReasonCode: "PSID_NOT_EXPOSED", Message: "The supported BF4 System and identity-selected chassis do not expose a PSID AssetTag.", Tool: "redfish"} + } + return inventory.Observation{State: "observed", Values: []string{value}, ReasonCode: "PSID_OBSERVED", Message: "The supported BF4 AssetTag PSID was verified against unchanged identity-bound System and chassis observations.", Tool: "redfish"} +} diff --git a/pkg/dpf/redfish/psid_test.go b/pkg/dpf/redfish/psid_test.go new file mode 100644 index 00000000..caf0ac7c --- /dev/null +++ b/pkg/dpf/redfish/psid_test.go @@ -0,0 +1,70 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package redfish + +import ( + "encoding/json" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("BF4 Redfish PSID observation", func() { + It("keeps System-first and fallback values while rejecting unsafe or contradictory values", func() { + const a, b = "MT_0000001774", "MT_0000001775" + for _, tc := range []struct { + name string + system, card Document + reason string + }{ + {"System value", Document{"AssetTag": a}, Document{}, "PSID_OBSERVED"}, + {"chassis fallback", Document{}, Document{"AssetTag": a}, "PSID_OBSERVED"}, + {"null System fallback", Document{"AssetTag": nil}, Document{"AssetTag": a}, "PSID_OBSERVED"}, + {"empty System fallback", Document{"AssetTag": ""}, Document{"AssetTag": a}, "PSID_OBSERVED"}, + {"N/A System fallback", Document{"AssetTag": "N/A"}, Document{"AssetTag": a}, "PSID_OBSERVED"}, + {"matching values", Document{"AssetTag": a}, Document{"AssetTag": a}, "PSID_OBSERVED"}, + {"different spelling", Document{"AssetTag": a}, Document{"AssetTag": strings.ToLower(a)}, "PSID_CONFLICT"}, + {"System with N/A chassis", Document{"AssetTag": a}, Document{"AssetTag": "N/A"}, "PSID_OBSERVED"}, + {"contradictory values", Document{"AssetTag": a}, Document{"AssetTag": b}, "PSID_CONFLICT"}, + {"malformed System type", Document{"AssetTag": 42}, Document{"AssetTag": a}, "PSID_INVALID"}, + {"malformed chassis type", Document{"AssetTag": a}, Document{"AssetTag": false}, "PSID_INVALID"}, + {"invalid control text", Document{"AssetTag": "credential-canary\n"}, Document{"AssetTag": a}, "PSID_INVALID"}, + {"invalid whitespace", Document{"AssetTag": " MT_0000001774"}, Document{}, "PSID_INVALID"}, + {"invalid oversized", Document{"AssetTag": strings.Repeat("A", 65)}, Document{}, "PSID_INVALID"}, + {"missing", Document{}, Document{}, "PSID_NOT_EXPOSED"}, + {"null", Document{"AssetTag": nil}, Document{"AssetTag": nil}, "PSID_NOT_EXPOSED"}, + {"sentinels", Document{"AssetTag": "N/A"}, Document{"AssetTag": ""}, "PSID_NOT_EXPOSED"}, + } { + By(tc.name) + o := bf4PSID(tc.system, tc.card, tc.system, tc.card) + Expect(o.ReasonCode).To(Equal(tc.reason)) + Expect(o.Tool).To(Equal("redfish")) + Expect(o.Message).NotTo(BeEmpty()) + if tc.reason == "PSID_OBSERVED" { + Expect(o.State).To(Equal("observed")) + Expect(o.Values).To(Equal([]string{a})) + } else { + Expect(o.State).NotTo(Equal("observed")) + Expect(o.Values).To(BeEmpty()) + } + encoded, err := json.Marshal(o) + Expect(err).NotTo(HaveOccurred()) + Expect(string(encoded)).NotTo(ContainSubstring("credential-canary")) + } + }) + It("invalidates initial values when either source field changes, including absence", func() { + for _, tc := range []struct{ system, card, currentSystem, currentCard Document }{ + {Document{"AssetTag": "MT_0000001774"}, Document{}, Document{"AssetTag": "MT_0000001775"}, Document{}}, + {Document{}, Document{"AssetTag": "MT_0000001774"}, Document{}, Document{"AssetTag": "N/A"}}, + {Document{}, Document{"AssetTag": "MT_0000001774"}, Document{"AssetTag": "MT_0000001774"}, Document{"AssetTag": "MT_0000001774"}}, + {Document{}, Document{"AssetTag": "MT_0000001774"}, Document{"AssetTag": nil}, Document{"AssetTag": "MT_0000001774"}}, + } { + o := bf4PSID(tc.system, tc.card, tc.currentSystem, tc.currentCard) + Expect(o.State).To(Equal("unavailable")) + Expect(o.ReasonCode).To(Equal("PSID_CHANGED")) + Expect(o.Values).To(BeEmpty()) + } + }) +}) diff --git a/pkg/dpf/redfish/refresh_integration_test.go b/pkg/dpf/redfish/refresh_integration_test.go new file mode 100644 index 00000000..ce397ea2 --- /dev/null +++ b/pkg/dpf/redfish/refresh_integration_test.go @@ -0,0 +1,115 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish_test + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/artifacts" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("host-only Redfish refresh ownership", func() { + It("rejects replacement at the same resource before publishing over the previous handoff", func() { + serial := "CARD-A" + link := func(uri string) redfish.Document { return redfish.Document{"@odata.id": uri} } + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var doc redfish.Document + switch r.URL.Path { + case "/redfish/v1/": + doc = redfish.Document{"SessionService": link("/redfish/v1/SessionService"), "Systems": link("/redfish/v1/Systems")} + case "/redfish/v1/SessionService": + doc = redfish.Document{"Sessions": link("/redfish/v1/SessionService/Sessions")} + case "/redfish/v1/SessionService/Sessions": + w.Header().Set("X-Auth-Token", "fixture") + w.Header().Set("Location", "/redfish/v1/SessionService/Sessions/owned") + w.WriteHeader(http.StatusCreated) + doc = redfish.Document{} + case "/redfish/v1/SessionService/Sessions/owned": + w.WriteHeader(http.StatusNoContent) + return + case "/redfish/v1/Systems": + doc = redfish.Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Systems/host")}} + case "/redfish/v1/Systems/host": + doc = redfish.Document{"UUID": "11111111-1111-4111-8111-111111111111", "Manufacturer": "Dell", "Model": "PowerEdge", "NetworkInterfaces": link("/redfish/v1/Systems/host/NetworkInterfaces")} + case "/redfish/v1/Systems/host/NetworkInterfaces": + doc = redfish.Document{"Members@odata.count": 1, "Members": []any{link("/redfish/v1/Systems/host/NetworkInterfaces/1")}} + case "/redfish/v1/Systems/host/NetworkInterfaces/1": + doc = redfish.Document{"Links": redfish.Document{"NetworkAdapter": link("/redfish/v1/Chassis/host/NetworkAdapters/slot1")}} + case "/redfish/v1/Chassis/host/NetworkAdapters/slot1": + doc = redfish.Document{"SerialNumber": serial, "Model": "BlueField-3"} + default: + w.WriteHeader(http.StatusNotFound) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(doc) + })) + DeferCleanup(server.Close) + dir, err := os.MkdirTemp("", "dpf-refresh-") + Expect(err).NotTo(HaveOccurred()) + dir, err = filepath.EvalSymlinks(dir) + Expect(err).NotTo(HaveOccurred()) + DeferCleanup(os.RemoveAll, dir) + Expect(os.Setenv("DPF_REFRESH_TEST_PASSWORD", "fixture")).To(Succeed()) + DeferCleanup(os.Unsetenv, "DPF_REFRESH_TEST_PASSWORD") + seed := fmt.Sprintf("credentials:\n reader: {username: reader, passwordRef: {env: DPF_REFRESH_TEST_PASSWORD}}\ndiscovery:\n tls: {insecureSkipVerify: true}\n hosts: [{bmcEndpoint: '%s', credentialRef: reader}]\n probes: {mode: inventory}\n", server.URL) + in, err := config.Decode([]byte(seed), filepath.Join(dir, "seed.yaml")) + Expect(err).NotTo(HaveOccurred()) + req, err := config.Resolve(in) + Expect(err).NotTo(HaveOccurred()) + observe := func() inventory.Inventory { + host, _, err := (redfish.Collector{}).Collect(context.Background(), req.Targets[0], time.Second) + Expect(err).NotTo(HaveOccurred()) + inv := inventory.Inventory{Collection: inventory.Collection{Mode: "inventory"}, Hosts: []inventory.Device{host}, RequestedScope: []inventory.ScopeEndpoint{{Role: "hosts", Endpoint: server.URL, TLSInsecureSkipVerify: true}}} + inv.DPUs = discovery.HostCards(inv.Hosts) + inv.Associations, _ = discovery.Correlate(inv.DPUs, inv.Hosts) + Expect(inv.DPUs).To(HaveLen(1)) + Expect(inv.Associations).To(HaveLen(1)) + discovery.Canonicalize(&inv) + return inv + } + old := observe() + for _, replacement := range []bool{false, true} { + if replacement { + serial = "CARD-B" + } + current := observe() + Expect(current.ScopeDigest).To(Equal(old.ScopeDigest)) + Expect(current.DPUs[0].RecordKey).To(Equal(old.DPUs[0].RecordKey)) + Expect(inventory.IdentityConflict(&old, ¤t)).To(Equal(replacement)) + output := filepath.Join(dir, fmt.Sprintf("discovered-%t.yaml", replacement)) + data, err := config.Saved(in, &old, output) + Expect(err).NotTo(HaveOccurred()) + Expect(os.WriteFile(output, data, 0600)).To(Succeed()) + prior, err := config.Load(output) + Expect(err).NotTo(HaveOccurred()) + store, err := artifacts.New(prior, output, true) + Expect(err).NotTo(HaveOccurred()) + report := discovery.NewReport() + report.Inventory = ¤t + report.Discovery.Outcome = "complete" + err = store.Publish(context.Background(), &report) + if replacement { + Expect(err).To(MatchError(ContainSubstring("physical identity"))) + unchanged, readErr := os.ReadFile(output) + Expect(readErr).NotTo(HaveOccurred()) + Expect(unchanged).To(Equal(data)) + } else { + Expect(err).NotTo(HaveOccurred()) + } + } + }) +}) diff --git a/pkg/dpf/redfish/software.go b/pkg/dpf/redfish/software.go new file mode 100644 index 00000000..59977913 --- /dev/null +++ b/pkg/dpf/redfish/software.go @@ -0,0 +1,328 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "errors" + "fmt" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +// missing and unavailable deliberately describe our observation, never a guessed +// device capability. Response bodies and error strings are not diagnostic inputs. +func missing(code, message string) inventory.Observation { + return inventory.Observation{State: "unknown", ReasonCode: code, Message: message} +} +func unavailable(err error) inventory.Observation { + reason := Code(err) + message := "The Redfish observation failed; inspect the reason code." + switch reason { + case "ACCESS_DENIED": + message = "The account is not authorized to read this Redfish resource." + case "AUTHENTICATION_FAILED": + message = "The Redfish service rejected authentication for this read." + case "RESOURCE_NOT_FOUND": + message = "The advertised Redfish resource was not found." + case "BMC_TIMEOUT": + message = "The Redfish read exceeded its timeout." + case "CANCELLED": + message = "The Redfish read was cancelled." + case "TRANSPORT_FAILED": + message = "The Redfish connection could not complete." + case "TLS_VERIFICATION_FAILED": + message = "The Redfish server certificate could not be verified." + case "INVALID_RESPONSE", "INCOMPLETE_COLLECTION", "RESPONSE_READ_FAILED": + message = "The Redfish response was malformed or incomplete." + case "RESOURCE_LIMIT", "RESPONSE_LIMIT", "PAGINATION_LIMIT": + message = "The Redfish read exceeded its resource, size or pagination limit." + } + var remote *Error + if errors.As(err, &remote) && remote.Status != 0 { + message += fmt.Sprintf(" HTTP %d.", remote.Status) + } + return inventory.Observation{State: "unavailable", ReasonCode: reason, Message: message} +} + +func collectSoftware(ctx context.Context, reader Reader, root Document, system Resource, device *inventory.Device) string { + if device.Observations == nil { + device.Observations = map[string]inventory.Observation{} + } + device.Observations["client-certificate-support"] = missing("NOT_EVALUATED", "Inventory collection does not test DPF client-certificate authentication.") + manager, managerState := softwareManager(ctx, reader, root) + device.Observations["managers"] = managerState + device.Time = inventory.TimeObservation{State: "unknown", ReasonCode: managerState.ReasonCode, Message: managerState.Message} + if managerState.State == "observed" { + collectManager(ctx, reader, manager, device) + } else { + device.Observations["bmc-time"], device.Observations["bmc-ntp"] = managerState, managerState + } + records, state := softwareInventory(ctx, reader, root) + device.Observations["firmware-inventory"] = state + for _, resource := range records { + firmware := firmwareRecord(resource, system.URI, manager.URI, device.BMCEndpoint) + device.Firmware = append(device.Firmware, firmware) + } + for _, component := range []string{"bmc", "cec", "atf", "uefi", "nic", "dpu-os", "dpu-bsp", "golden"} { + found := false + for _, f := range device.Firmware { + found = found || f.Component == component + } + if !found { + reason := state + if state.State == "observed" { + reason = missing("COMPONENT_NOT_EXPOSED", "The firmware inventory does not expose this component.") + } + device.Firmware = append(device.Firmware, inventory.Firmware{Component: component, State: "unknown", ReasonCode: reason.ReasonCode, Message: reason.Message, Activation: "unknown", ActivationReasonCode: "VERSION_UNAVAILABLE", ActivationMessage: "Activation cannot be established without the component observation."}) + } + } + return reconcileFirmware(device) +} + +func softwareManager(ctx context.Context, reader Reader, root Document) (Resource, inventory.Observation) { + uri := Link(root, "Managers") + if uri == "" { + return Resource{}, missing("LINK_NOT_EXPOSED", "The service root does not advertise Managers.") + } + managers, err := reader.Collection(ctx, uri) + if err != nil { + return Resource{}, unavailable(err) + } + if len(managers) != 1 { + return Resource{}, missing("MANAGER_SELECTION_AMBIGUOUS", "A unique BMC manager could not be selected; manager firmware and time remain unknown.") + } + return managers[0], inventory.Observation{State: "observed"} +} + +func collectManager(ctx context.Context, reader Reader, manager Resource, device *inventory.Device) { + if version := Text(manager.Document, "FirmwareVersion"); strings.TrimSpace(version) != "" { + device.Firmware = append(device.Firmware, inventory.Firmware{ResourceURI: manager.URI, Component: "bmc", State: "known", RawVersion: version, Activation: "unknown", ActivationReasonCode: "ACTIVATION_NOT_EXPOSED", ActivationMessage: "Manager firmware metadata does not identify an active software inventory image."}) + } + datetime := Text(manager.Document, "DateTime") + state := missing("PROPERTY_NOT_EXPOSED", "The manager does not expose DateTime.") + if value, exists := manager.Document["DateTime"]; exists && value != nil && datetime == "" { + state = missing("INVALID_DATETIME", "The manager DateTime is not a nonempty RFC3339 timestamp.") + } + if datetime != "" { + if _, err := time.Parse(time.RFC3339, datetime); err != nil { + state = missing("INVALID_DATETIME", "The manager DateTime is not a valid RFC3339 timestamp.") + } else { + device.Time.DateTime, device.Time.Offset = datetime, Text(manager.Document, "DateTimeLocalOffset") + state = inventory.Observation{State: "observed"} + } + } + device.Time.State, device.Time.ReasonCode, device.Time.Message = state.State, state.ReasonCode, state.Message + if state.State == "observed" { + device.Time.State = "known" + } + device.Observations["bmc-time"] = state + uri := Link(manager.Document, "NetworkProtocol") + state = missing("LINK_NOT_EXPOSED", "The manager does not advertise NetworkProtocol; NTP configuration is unknown.") + if uri != "" { + np, err := reader.Get(ctx, uri) + if err != nil { + state = unavailable(err) + } else { + ntp := Object(np, "NTP") + enabled, hasEnabled := ntp["ProtocolEnabled"].(bool) + servers, hasServers := ntp["NTPServers"].([]any) + validServers := hasServers + for _, value := range servers { + server, ok := value.(string) + if !ok { + validServers = false + break + } + device.Time.NTPServers = append(device.Time.NTPServers, server) + } + if !validServers { + device.Time.NTPServers = nil + } + if hasEnabled { + device.Time.NTPEnabled = &enabled + } + state = missing("PROPERTY_NOT_EXPOSED", "NTP enablement or server configuration is not exposed.") + if (ntp["ProtocolEnabled"] != nil && !hasEnabled) || (ntp["NTPServers"] != nil && !validServers) { + state = missing("PROPERTY_INVALID", "The NTP enablement or server configuration has an invalid type.") + } + if hasEnabled && validServers { + state = inventory.Observation{State: "observed"} + } + } + } + device.Observations["bmc-ntp"] = state +} + +func softwareInventory(ctx context.Context, reader Reader, root Document) ([]Resource, inventory.Observation) { + uri := Link(root, "UpdateService") + if uri == "" { + return nil, missing("LINK_NOT_EXPOSED", "The service root does not advertise UpdateService.") + } + update, err := reader.Get(ctx, uri) + if err != nil { + return nil, unavailable(err) + } + uri = Link(update, "FirmwareInventory") + if uri == "" { + return nil, missing("LINK_NOT_EXPOSED", "UpdateService does not advertise FirmwareInventory.") + } + records, err := reader.Collection(ctx, uri) + if err != nil { + return records, unavailable(err) + } + return records, inventory.Observation{State: "observed"} +} + +func componentName(id, name string) string { + normalize := func(s string) string { + return strings.Trim(strings.Map(func(r rune) rune { + if r >= 'a' && r <= 'z' || r >= '0' && r <= '9' { + return r + } + return '-' + }, strings.ToLower(s)), "-") + } + // Prefer a recognizable stable ID, then the display name; golden must precede + // component aliases because a golden BMC image is not the running BMC image. + // Keep explicitly identified BMC and CPU ERoTs separate from each other + // and from BMC firmware. An ERoT name alone does not identify CEC firmware. + for _, raw := range []string{id, name} { + s := normalize(raw) + words := "-" + s + "-" + switch { + case strings.Contains(words, "-golden-"): + return "golden" + case strings.Contains(words, "-erot-") && strings.Contains(words, "-bmc-") && !strings.Contains(words, "-cpu-"): + return "bmc-erot" + case strings.Contains(words, "-erot-") && strings.Contains(words, "-cpu-") && !strings.Contains(words, "-bmc-"): + return "cpu-erot" + case strings.Contains(words, "-erot-"): + return "erot" + case strings.Contains(words, "-bmc-"): + return "bmc" + case strings.Contains(words, "-cec-"): + return "cec" + case strings.Contains(words, "-atf-") && strings.Contains(words, "-uefi-"): + return "atf-uefi" + case strings.Contains(words, "-atf-"): + return "atf" + case strings.Contains(words, "-uefi-"): + return "uefi" + case strings.Contains(words, "-nic-"): + return "nic" + case s == "dpu-os" || s == "dpu-operating-system": + return "dpu-os" + case s == "dpu-bsp" || s == "bsp": + return "dpu-bsp" + } + } + return "other" +} + +func firmwareRecord(resource Resource, system, manager, origin string) inventory.Firmware { + f := inventory.Firmware{ResourceURI: resource.URI, RawID: Text(resource.Document, "Id"), RawName: Text(resource.Document, "Name"), RawVersion: Text(resource.Document, "Version"), State: "known", Activation: "unknown", ActivationReasonCode: "ACTIVATION_NOT_EXPOSED", ActivationMessage: "No explicit active or staged target link identifies the selected device."} + f.Component = componentName(f.RawID, f.RawName) + if strings.TrimSpace(f.RawVersion) == "" { + f.State, f.ReasonCode, f.Message = "unknown", "VERSION_NOT_EXPOSED", "This firmware entry does not expose a nonempty Version." + if value := resource.Document["Version"]; value != nil { + if _, ok := value.(string); !ok { + f.ReasonCode, f.Message = "VERSION_INVALID", "The firmware Version property is not a string." + } + } + } else if f.Component == "other" { + f.ReasonCode, f.Message = "COMPONENT_UNRECOGNIZED", "The raw firmware identity is retained; its component is not recognized." + } + // DMTF SoftwareInventory v1.12 Links.ActiveTargets/StagedTargets describe + // actual activation, unlike Updateable, Status.Enabled, or an image's name. + links := Object(resource.Document, "Links") + // Reuse the transport's origin/path rules without performing a request. + // Target identity links cannot carry even the pagination queries accepted + // by the general reader. Normalize trailing slash only after validation. + validator := Client{origin: origin} + normalize := func(ref string) (string, error) { + uri, err := validator.guard(ref) + if err != nil { + return "", err + } + if strings.Contains(uri, "?") { + return "", &Error{Code: "UNSAFE_QUERY"} + } + return strings.TrimSuffix(uri, "/"), nil + } + selected := map[string]bool{} + for _, ref := range []string{system, manager} { + if ref == "" { + continue + } + uri, err := normalize(ref) + if err != nil { + f.ActivationReasonCode, f.ActivationMessage = Code(err), "The selected resource link is invalid; firmware activation cannot be established." + return f + } + selected[uri] = true + } + matches := func(key string) (bool, error) { + value, exists := links[key] + if !exists || value == nil { + return false, nil + } + targets, ok := value.([]any) + if !ok { + return false, &Error{Code: "ACTIVATION_LINK_INVALID"} + } + found := false + for _, target := range targets { + uri, err := normalize(Link(Document{"target": target}, "target")) + if err != nil { + return false, err + } + found = found || selected[uri] + } + return found, nil + } + active, activeErr := matches("ActiveTargets") + staged, stagedErr := matches("StagedTargets") + if err := errors.Join(activeErr, stagedErr); err != nil { + f.ActivationReasonCode, f.ActivationMessage = Code(err), "ActiveTargets or StagedTargets contains an invalid or unsafe resource link; activation remains unknown." + return f + } + if active != staged { + f.Activation = "running" + if staged { + f.Activation = "staged" + } + f.ActivationReasonCode, f.ActivationMessage = "", "" + } else if active { + f.ActivationReasonCode, f.ActivationMessage = "ACTIVATION_CONFLICT", "The selected target is reported as both active and staged." + } + return f +} + +func reconcileFirmware(device *inventory.Device) string { + versions := map[string]map[string]bool{} + for _, f := range device.Firmware { + // Generic identities cannot establish that entries describe one component. + if f.State != "known" || f.Activation == "staged" || f.Component == "other" || f.Component == "erot" { + continue + } + if versions[f.Component] == nil { + versions[f.Component] = map[string]bool{} + } + versions[f.Component][strings.TrimSpace(f.RawVersion)] = true + } + bmc := "" + for i := range device.Firmware { + f := &device.Firmware[i] + if len(versions[f.Component]) > 1 && f.Activation != "staged" { + f.State, f.ReasonCode, f.Message = "conflict", "FIRMWARE_VERSION_CONFLICT", "Sources report different versions for this component without an explicit staged-image distinction." + device.Observations["firmware."+f.Component] = missing(f.ReasonCode, f.Message) + } + if f.Component == "bmc" && f.State == "known" && f.Activation != "staged" { + bmc = f.RawVersion + } + } + return bmc +} diff --git a/pkg/dpf/redfish/software_test.go b/pkg/dpf/redfish/software_test.go new file mode 100644 index 00000000..9a416bb0 --- /dev/null +++ b/pkg/dpf/redfish/software_test.go @@ -0,0 +1,262 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package redfish + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +type softwareReader struct { + docs map[string]Document + collections map[string][]Resource + failures map[string]error +} + +func (r softwareReader) Get(_ context.Context, uri string) (Document, error) { + return r.docs[uri], r.failures[uri] +} +func (r softwareReader) Collection(_ context.Context, uri string) ([]Resource, error) { + return r.collections[uri], r.failures[uri] +} +func softwareFixture() (softwareReader, Document, Resource) { + return softwareReader{ + docs: map[string]Document{"/redfish/v1/UpdateService": {"FirmwareInventory": Document{"@odata.id": "/redfish/v1/UpdateService/FirmwareInventory"}}, "/redfish/v1/Managers/BMC/NetworkProtocol": {"NTP": Document{"ProtocolEnabled": false, "NTPServers": []any{}}}}, + collections: map[string][]Resource{"/redfish/v1/Managers": {{URI: "/redfish/v1/Managers/BMC", Document: Document{"FirmwareVersion": "1.0", "DateTime": "2026-11-06T12:00:00Z", "NetworkProtocol": Document{"@odata.id": "/redfish/v1/Managers/BMC/NetworkProtocol"}}}}}, + failures: map[string]error{}, + }, Document{"Managers": Document{"@odata.id": "/redfish/v1/Managers"}, "UpdateService": Document{"@odata.id": "/redfish/v1/UpdateService"}}, Resource{URI: "/redfish/v1/Systems/DPU", Document: Document{}} +} + +var _ = Describe("optional Redfish software observations", func() { + It("classifies component IDs and keeps raw versions and resource provenance", func() { + for id, expected := range map[string]string{"BMC_Firmware": "bmc", "Bluefield_BMC_ERoT_FW": "bmc-erot", "Bluefield_CPU_ERoT_FW": "cpu-erot", "ERoT_Firmware": "erot", "DPU_OS": "dpu-os", "DPU_BSP": "dpu-bsp", "CEC_Firmware": "cec", "DPU_ATF": "atf", "DPU_UEFI": "uefi", "NIC_Firmware": "nic", "BMC_Golden_Image": "golden", "unrecognized": "other"} { + f := firmwareRecord(Resource{URI: "/redfish/v1/UpdateService/FirmwareInventory/" + id, Document: Document{"Id": id, "Name": "Inventory entry", "Version": " v1.2 "}}, "", "", "https://bmc.invalid") + Expect(f.Component).To(Equal(expected), id) + Expect(f.RawID).To(Equal(id)) + Expect(f.RawVersion).To(Equal(" v1.2 ")) + Expect(f.EvidenceRefs).To(BeNil()) + Expect(f.Activation).To(Equal("unknown")) + Expect(f.ActivationReasonCode).To(Equal("ACTIVATION_NOT_EXPOSED")) + Expect(f.ActivationMessage).NotTo(BeEmpty()) + } + }) + It("requires explicit target links for activation and does not infer it from names or enabled status", func() { + doc := Document{"Id": "DPU_OS", "Name": "Active image", "Version": "1", "Status": Document{"State": "Enabled"}, "Updateable": true} + record := Resource{URI: "/redfish/v1/UpdateService/FirmwareInventory/os", Document: doc} + Expect(firmwareRecord(record, "/redfish/v1/Systems/selected", "", "https://bmc.invalid").Activation).To(Equal("unknown")) + doc["Links"] = Document{"ActiveTargets": []any{Document{"@odata.id": "/redfish/v1/Systems/other"}}} + Expect(firmwareRecord(record, "/redfish/v1/Systems/selected", "", "https://bmc.invalid").Activation).To(Equal("unknown")) + doc["Links"] = Document{"StagedTargets": []any{Document{"@odata.id": "/redfish/v1/Systems/selected"}}} + Expect(firmwareRecord(record, "/redfish/v1/Systems/selected", "", "https://bmc.invalid").Activation).To(Equal("staged")) + doc["Links"] = Document{"ActiveTargets": []any{Document{"@odata.id": "/redfish/v1/Systems/selected"}}} + Expect(firmwareRecord(record, "/redfish/v1/Systems/selected", "", "https://bmc.invalid").Activation).To(Equal("running")) + Object(doc, "Links")["StagedTargets"] = []any{Document{"@odata.id": "/redfish/v1/Systems/selected"}} + f := firmwareRecord(record, "/redfish/v1/Systems/selected", "", "https://bmc.invalid") + Expect(f.Activation).To(Equal("unknown")) + Expect(f.ActivationReasonCode).To(Equal("ACTIVATION_CONFLICT")) + }) + It("normalizes validated same-origin activation targets and keeps staged versions separate", func() { + reader, root, system := softwareFixture() + device := inventory.Device{BMCEndpoint: "https://bmc.invalid"} + reader.collections["/redfish/v1/UpdateService/FirmwareInventory"] = []Resource{ + {URI: "/redfish/v1/UpdateService/FirmwareInventory/nic-current", Document: Document{"Id": "NIC_Firmware", "Version": "1.0", "Links": Document{"ActiveTargets": []any{Document{"@odata.id": "https://bmc.invalid/redfish/v1/Systems/DPU/"}}}}}, + {URI: "/redfish/v1/UpdateService/FirmwareInventory/nic-next", Document: Document{"Id": "NIC_Firmware", "Version": "2.0", "Links": Document{"StagedTargets": []any{Document{"@odata.id": "https://bmc.invalid/redfish/v1/Systems/DPU"}}}}}, + } + collectSoftware(context.Background(), reader, root, system, &device) + count := 0 + for _, f := range device.Firmware { + if f.Component != "nic" { + continue + } + count++ + Expect(f.State).To(Equal("known")) + Expect(f.ReasonCode).To(BeEmpty()) + Expect(f.ActivationReasonCode).To(BeEmpty()) + if f.RawVersion == "1.0" { + Expect(f.Activation).To(Equal("running")) + } else { + Expect(f.Activation).To(Equal("staged")) + } + } + Expect(count).To(Equal(2)) + Expect(device.Observations).NotTo(HaveKey("firmware.nic")) + for _, ref := range []string{"/redfish/v1/Systems/DPU/", "https://bmc.invalid:443/redfish/v1/Systems/DPU", "https://BMC.INVALID/redfish/v1/Systems/DPU/"} { + f := firmwareRecord(Resource{Document: Document{"Id": "NIC_Firmware", "Version": "1", "Links": Document{"ActiveTargets": []any{Document{"@odata.id": ref}}}}}, "/redfish/v1/Systems/DPU/", "", "https://bmc.invalid") + Expect(f.Activation).To(Equal("running"), ref) + } + f := firmwareRecord(Resource{Document: Document{"Id": "BMC", "Version": "1", "Links": Document{"ActiveTargets": []any{Document{"@odata.id": "https://bmc.invalid/redfish/v1/Managers/BMC/"}}}}}, "/redfish/v1/Systems/DPU", "/redfish/v1/Managers/BMC/", "https://bmc.invalid") + Expect(f.Activation).To(Equal("running")) + }) + It("never promotes foreign, malformed or query-bearing activation links", func() { + for _, scenario := range []struct { + value any + code string + }{ + {[]any{Document{"@odata.id": "https://foreign.invalid/redfish/v1/Systems/DPU"}}, "CROSS_ORIGIN_LINK"}, + {[]any{Document{"@odata.id": "//bmc.invalid/redfish/v1/Systems/DPU"}}, "CROSS_ORIGIN_LINK"}, + {[]any{Document{"@odata.id": "https://reader:secret@bmc.invalid/redfish/v1/Systems/DPU"}}, "UNSAFE_LINK"}, + {[]any{Document{"@odata.id": "/redfish/v1/Systems/DPU/../DPU"}}, "UNSAFE_LINK"}, + {[]any{Document{"@odata.id": "/redfish/v1/Systems/%44PU"}}, "UNSAFE_LINK"}, + {[]any{Document{"@odata.id": "/redfish/v1/Systems/DPU#fragment"}}, "UNSAFE_LINK"}, + {[]any{Document{"@odata.id": "/redfish/v1/Systems/DPU?$skip=0"}}, "UNSAFE_QUERY"}, + {[]any{Document{"@odata.id": "/redfish/v1/Systems/DPU?"}}, "UNSAFE_QUERY"}, + {[]any{Document{"@odata.id": 123}}, "UNSAFE_LINK"}, + {Document{"@odata.id": "/redfish/v1/Systems/DPU"}, "ACTIVATION_LINK_INVALID"}, + } { + // A valid active member must not hide a malformed or foreign staged + // member; otherwise incomplete activation evidence could pass a BOM. + doc := Document{"Id": "NIC_Firmware", "Version": "1", "Links": Document{"ActiveTargets": []any{Document{"@odata.id": "/redfish/v1/Systems/DPU"}}, "StagedTargets": scenario.value}} + f := firmwareRecord(Resource{Document: doc}, "/redfish/v1/Systems/DPU", "", "https://bmc.invalid") + Expect(f.Activation).To(Equal("unknown")) + Expect(f.ActivationReasonCode).To(Equal(scenario.code)) + Expect(f.ActivationMessage).NotTo(BeEmpty()) + Expect(f.ActivationMessage).NotTo(ContainSubstring("secret")) + } + doc := Document{"Id": "NIC_Firmware", "Version": "1", "Links": Document{"ActiveTargets": []any{Document{"@odata.id": "/redfish/v1/Systems/DPU"}, Document{"@odata.id": "https://foreign.invalid/redfish/v1/Systems/DPU"}}}} + f := firmwareRecord(Resource{Document: doc}, "/redfish/v1/Systems/DPU", "", "https://bmc.invalid") + Expect(f.Activation).To(Equal("unknown")) + Expect(f.ActivationReasonCode).To(Equal("CROSS_ORIGIN_LINK")) + }) + It("retains a denied source reason and does not equate it with an absent property", func() { + reader, root, system := softwareFixture() + reader.failures["/redfish/v1/UpdateService/FirmwareInventory"] = &Error{Code: "ACCESS_DENIED", Status: 403} + reader.failures["/redfish/v1/Managers/BMC/NetworkProtocol"] = &Error{Code: "ACCESS_DENIED", Status: 403} + d := inventory.Device{} + collectSoftware(context.Background(), reader, root, system, &d) + Expect(d.Time.DateTime).To(Equal("2026-11-06T12:00:00Z")) + Expect(d.Observations["bmc-ntp"].ReasonCode).To(Equal("ACCESS_DENIED")) + Expect(d.Observations["bmc-ntp"].Message).To(ContainSubstring("403")) + for _, f := range d.Firmware { + if f.Component != "bmc" { + Expect(f.ReasonCode).To(Equal("ACCESS_DENIED")) + Expect(f.Message).NotTo(BeEmpty()) + } + } + }) + It("preserves available entries in an incomplete firmware collection", func() { + reader, root, system := softwareFixture() + reader.collections["/redfish/v1/UpdateService/FirmwareInventory"] = []Resource{{URI: "/redfish/v1/UpdateService/FirmwareInventory/os", Document: Document{"Id": "DPU_OS", "Version": "24.04"}}} + reader.failures["/redfish/v1/UpdateService/FirmwareInventory"] = &Error{Code: "ACCESS_DENIED", Status: 403} + d := inventory.Device{} + collectSoftware(context.Background(), reader, root, system, &d) + Expect(d.Observations["firmware-inventory"].State).To(Equal("unavailable")) + Expect(d.Firmware).To(ContainElement(And(HaveField("Component", "dpu-os"), HaveField("RawVersion", "24.04"), HaveField("State", "known")))) + }) + It("retains conflicting manager and component versions instead of selecting one", func() { + reader, root, system := softwareFixture() + reader.collections["/redfish/v1/UpdateService/FirmwareInventory"] = []Resource{{URI: "/redfish/v1/UpdateService/FirmwareInventory/bmc", Document: Document{"Id": "BMC_Firmware", "Version": "2.0"}}} + d := inventory.Device{} + selected := collectSoftware(context.Background(), reader, root, system, &d) + Expect(selected).To(BeEmpty()) + Expect(d.Observations["firmware.bmc"].ReasonCode).To(Equal("FIRMWARE_VERSION_CONFLICT")) + count := 0 + for _, f := range d.Firmware { + if f.Component == "bmc" { + count++ + Expect(f.State).To(Equal("conflict")) + Expect(f.RawVersion).NotTo(BeEmpty()) + } + } + Expect(count).To(Equal(2)) + }) + It("keeps BMC and CPU ERoT versions distinct without conflating unidentified roots of trust", func() { + reader, root, system := softwareFixture() + reader.collections["/redfish/v1/UpdateService/FirmwareInventory"] = []Resource{ + {URI: "/redfish/v1/UpdateService/FirmwareInventory/BMC", Document: Document{"Id": "BMC_Firmware", "Version": "1.0"}}, + {URI: "/redfish/v1/UpdateService/FirmwareInventory/BMC_ERoT", Document: Document{"Id": "Bluefield_BMC_ERoT_FW", "Name": "BMC Firmware", "Version": "2.0"}}, + {URI: "/redfish/v1/UpdateService/FirmwareInventory/CPU_ERoT", Document: Document{"Id": "Bluefield_CPU_ERoT_FW", "Version": "3.0"}}, + {URI: "/redfish/v1/UpdateService/FirmwareInventory/ERoT_1", Document: Document{"Id": "ERoT_1", "Version": "4.0"}}, + {URI: "/redfish/v1/UpdateService/FirmwareInventory/ERoT_2", Document: Document{"Id": "ERoT_2", "Version": "5.0"}}, + } + d := inventory.Device{} + Expect(collectSoftware(context.Background(), reader, root, system, &d)).To(Equal("1.0")) + Expect(d.Observations).NotTo(HaveKey("firmware.bmc")) + Expect(d.Firmware).NotTo(ContainElement(HaveField("State", "conflict"))) + Expect(d.Firmware).To(ContainElement(And(HaveField("Component", "bmc-erot"), HaveField("RawVersion", "2.0"), HaveField("State", "known")))) + Expect(d.Firmware).To(ContainElement(And(HaveField("Component", "cpu-erot"), HaveField("RawVersion", "3.0"), HaveField("State", "known")))) + Expect(d.Firmware).To(ContainElement(And(HaveField("Component", "erot"), HaveField("RawVersion", "4.0"), HaveField("State", "known")))) + Expect(d.Firmware).To(ContainElement(And(HaveField("Component", "erot"), HaveField("RawVersion", "5.0"), HaveField("State", "known")))) + Expect(d.Firmware).To(ContainElement(And(HaveField("Component", "cec"), HaveField("RawVersion", ""), HaveField("ReasonCode", "COMPONENT_NOT_EXPOSED")))) + }) + It("keeps an explicitly staged successor separate from the manager version", func() { + reader, root, system := softwareFixture() + reader.collections["/redfish/v1/UpdateService/FirmwareInventory"] = []Resource{{URI: "/redfish/v1/UpdateService/FirmwareInventory/bmc", Document: Document{"Id": "BMC_Firmware", "Version": "2.0", "Links": Document{"StagedTargets": []any{Document{"@odata.id": "/redfish/v1/Managers/BMC"}}}}}} + d := inventory.Device{} + Expect(collectSoftware(context.Background(), reader, root, system, &d)).To(Equal("1.0")) + Expect(d.Firmware).To(ContainElement(And(HaveField("Component", "bmc"), HaveField("RawVersion", "2.0"), HaveField("Activation", "staged")))) + }) + It("does not select the first of multiple managers and preserves explicit false NTP", func() { + reader, root, system := softwareFixture() + d := inventory.Device{} + collectSoftware(context.Background(), reader, root, system, &d) + Expect(d.Time.NTPEnabled).NotTo(BeNil()) + Expect(*d.Time.NTPEnabled).To(BeFalse()) + Expect(d.Observations["bmc-ntp"].State).To(Equal("observed")) + reader.collections["/redfish/v1/Managers"] = append(reader.collections["/redfish/v1/Managers"], Resource{URI: "/redfish/v1/Managers/other", Document: Document{}}) + d = inventory.Device{} + Expect(collectSoftware(context.Background(), reader, root, system, &d)).To(BeEmpty()) + Expect(d.Time.ReasonCode).To(Equal("MANAGER_SELECTION_AMBIGUOUS")) + Expect(d.Time.Message).NotTo(BeEmpty()) + }) + It("keeps missing links, missing versions, invalid clock and transport errors distinct", func() { + reader, _, system := softwareFixture() + d := inventory.Device{} + collectSoftware(context.Background(), reader, Document{}, system, &d) + Expect(d.Observations["firmware-inventory"].ReasonCode).To(Equal("LINK_NOT_EXPOSED")) + f := firmwareRecord(Resource{URI: "/redfish/v1/UpdateService/FirmwareInventory/os", Document: Document{"Id": "DPU_OS"}}, "", "", "https://bmc.invalid") + Expect(f.ReasonCode).To(Equal("VERSION_NOT_EXPOSED")) + reader, root, system := softwareFixture() + reader.collections["/redfish/v1/Managers"][0].Document["DateTime"] = "invalid" + d = inventory.Device{} + collectSoftware(context.Background(), reader, root, system, &d) + Expect(d.Time.ReasonCode).To(Equal("INVALID_DATETIME")) + Expect(d.Time.DateTime).To(BeEmpty()) + for _, err := range []error{context.DeadlineExceeded, context.Canceled, &Error{Code: "RESOURCE_NOT_FOUND", Status: 404}, errors.New("secret-response")} { + state := unavailable(err) + Expect(state.ReasonCode).NotTo(BeEmpty()) + Expect(state.Message).NotTo(BeEmpty()) + Expect(state.Message).NotTo(ContainSubstring("secret-response")) + } + }) + It("returns already read members with a later collection error", func() { + first := "/redfish/v1/UpdateService/FirmwareInventory/first" + second := "/redfish/v1/UpdateService/FirmwareInventory/second" + collection := "/redfish/v1/UpdateService/FirmwareInventory" + c, closeServer := graphClient(map[string]Document{ + collection: {"Members@odata.count": 2, "Members": []any{Document{"@odata.id": first}, Document{"@odata.id": second}}}, + first: {"Id": "DPU_OS", "Version": "24.04"}, + }, nil) + defer closeServer() + records, err := c.Collection(context.Background(), collection) + Expect(Code(err)).To(Equal("RESOURCE_NOT_FOUND")) + Expect(records).To(HaveLen(1)) + Expect(records[0].URI).To(Equal(first)) + }) + It("distinguishes incomplete response bodies from response size limits", func() { + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Length", "100") + w.WriteHeader(200) + _, _ = w.Write([]byte("{}")) + })) + defer server.Close() + c := &Client{origin: server.URL, http: server.Client()} + _, err := c.Get(context.Background(), "/redfish/v1/Managers") + Expect(Code(err)).To(Equal("RESPONSE_READ_FAILED")) + }) + It("classifies an HTTP client timeout separately from other transport errors", func() { + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { time.Sleep(50 * time.Millisecond); w.WriteHeader(200) })) + defer server.Close() + client := server.Client() + client.Timeout = time.Millisecond + c := &Client{origin: server.URL, http: client} + _, err := c.Get(context.Background(), "/redfish/v1/Managers") + Expect(Code(err)).To(Equal("BMC_TIMEOUT")) + }) +}) diff --git a/pkg/dpf/releases/releases.go b/pkg/dpf/releases/releases.go new file mode 100644 index 00000000..9571fa46 --- /dev/null +++ b/pkg/dpf/releases/releases.go @@ -0,0 +1,230 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package releases owns the embedded, release-specific DPF software bill of +// materials. Catalog entries describe software coordinates and documented +// requirements; they do not qualify hardware or authorize installation. +package releases + +import ( + "bytes" + _ "embed" + "fmt" + "io" + "slices" + "strings" + + "gopkg.in/yaml.v3" +) + +// DefaultRelease is deliberately pinned. Adding a catalog entry never selects it +// automatically; promoting the default is a separate reviewed change. +const DefaultRelease = "26.4.0" + +//go:embed releases.yaml +var catalogYAML []byte + +// Source identifies the authoritative upstream material used to curate a BOM. +// Revision is a Git commit or the digest of a downloaded release artifact. +type Source struct { + ID string `json:"id" yaml:"id"` + URL string `json:"url" yaml:"url"` + Revision string `json:"revision" yaml:"revision"` +} + +type Operator struct { + Chart string `json:"chart" yaml:"chart"` + ChartRepository string `json:"chartRepository" yaml:"chartRepository"` + Version string `json:"version" yaml:"version"` + ChartDigest string `json:"chartDigest" yaml:"chartDigest"` + Image string `json:"image" yaml:"image"` + SourceRef string `json:"sourceRef" yaml:"sourceRef"` +} + +// Component is a release coordinate, not evidence that the component is present. +// Applicability distinguishes base components from conditional and optional ones. +type Component struct { + Name string `json:"name" yaml:"name"` + Version string `json:"version" yaml:"version"` + Repository string `json:"repository" yaml:"repository"` + Applicability string `json:"applicability" yaml:"applicability"` + SourceRef string `json:"sourceRef" yaml:"sourceRef"` +} + +type Artifact struct { + Version string `json:"version" yaml:"version"` + URL string `json:"url" yaml:"url"` + Digest string `json:"digest,omitempty" yaml:"digest,omitempty"` + ReasonCode string `json:"reasonCode,omitempty" yaml:"reasonCode,omitempty"` + Message string `json:"message,omitempty" yaml:"message,omitempty"` + SourceRef string `json:"sourceRef" yaml:"sourceRef"` +} + +// Requirement distinguishes a minimum capability version from a desired target. +// Unknown requirements have no numeric constraint and retain a reason. Scope +// identifies where a requirement applies, not where it must be measured. +type Requirement struct { + Component string `json:"component" yaml:"component"` + Scope string `json:"scope" yaml:"scope"` + State string `json:"state" yaml:"state"` + Minimum string `json:"minimum,omitempty" yaml:"minimum,omitempty"` + Target string `json:"target,omitempty" yaml:"target,omitempty"` + ReasonCode string `json:"reasonCode,omitempty" yaml:"reasonCode,omitempty"` + Message string `json:"message,omitempty" yaml:"message,omitempty"` + SourceRef string `json:"sourceRef" yaml:"sourceRef"` +} + +type HardwareBOM struct { + Family string `json:"family" yaml:"family"` + Models []string `json:"models" yaml:"models"` + BFB Artifact `json:"bfb" yaml:"bfb"` + Requirements []Requirement `json:"requirements" yaml:"requirements"` +} + +type Release struct { + Version string `json:"version" yaml:"version"` + Operator Operator `json:"operator" yaml:"operator"` + Components []Component `json:"components" yaml:"components"` + Hardware []HardwareBOM `json:"hardware" yaml:"hardware"` + Sources []Source `json:"sources" yaml:"sources"` + Limitations []string `json:"limitations" yaml:"limitations"` +} + +type catalog struct { + Releases map[string]Release `yaml:"releases"` +} + +// Resolve returns a fresh BOM with no shared mutable values. It performs no +// network access. Empty means the pinned default; callers with presence-aware +// configuration must reject a supplied empty value before calling Resolve. +func Resolve(version string) (Release, error) { + if version == "" { + version = DefaultRelease + } + c, err := load(catalogYAML) + if err != nil { + return Release{}, err + } + r, ok := c.Releases[version] + if !ok { + return Release{}, fmt.Errorf("unknown DPF release %q; catalog releases: %s", version, strings.Join(keys(c), ", ")) + } + return r, nil +} + +// SupportedReleases returns catalog keys. Presence does not assert runtime +// qualification for every component, hardware model or deployment profile. +func SupportedReleases() []string { + c, err := load(catalogYAML) + if err != nil { + return nil + } + return keys(c) +} + +func keys(c catalog) []string { + out := make([]string, 0, len(c.Releases)) + for k := range c.Releases { + out = append(out, k) + } + slices.Sort(out) + return out +} + +// RequirementsFor returns a copy of the hardware BOM for an exact documented +// model. Generic family names, unknown models and nil receivers do not match; +// callers must retain an explicit applicability-unresolved result. +func (r *Release) RequirementsFor(model string) (HardwareBOM, bool) { + if r == nil { + return HardwareBOM{}, false + } + model = strings.TrimSpace(model) + for _, h := range r.Hardware { + for _, supported := range h.Models { + if strings.EqualFold(model, supported) { + h.Models = slices.Clone(h.Models) + h.Requirements = slices.Clone(h.Requirements) + return h, true + } + } + } + return HardwareBOM{}, false +} + +func load(data []byte) (catalog, error) { + var c catalog + dec := yaml.NewDecoder(bytes.NewReader(data)) + dec.KnownFields(true) + if err := dec.Decode(&c); err != nil { + return catalog{}, fmt.Errorf("invalid embedded DPF release catalog: %w", err) + } + var trailing any + if err := dec.Decode(&trailing); err != io.EOF { + return catalog{}, fmt.Errorf("DPF release catalog must contain one document") + } + if len(c.Releases) == 0 { + return catalog{}, fmt.Errorf("DPF release catalog has no entries") + } + for key, r := range c.Releases { + if err := validate(key, r); err != nil { + return catalog{}, fmt.Errorf("invalid DPF release %q: %w", key, err) + } + } + return c, nil +} + +func validate(key string, r Release) error { + if r.Version != key || r.Operator.Version != "v"+key || r.Operator.Chart == "" || r.Operator.ChartRepository == "" || r.Operator.Image == "" || r.Operator.ChartDigest == "" { + return fmt.Errorf("missing or inconsistent release coordinates") + } + sources := map[string]bool{} + for _, s := range r.Sources { + if s.ID == "" || sources[s.ID] || !strings.HasPrefix(s.URL, "https://") || s.Revision == "" { + return fmt.Errorf("missing or duplicate authoritative source") + } + sources[s.ID] = true + } + if !sources[r.Operator.SourceRef] { + return fmt.Errorf("operator source is not declared") + } + components := map[string]bool{} + for _, c := range r.Components { + if c.Name == "" || components[c.Name] || c.Version == "" || c.Repository == "" || !sources[c.SourceRef] || !slices.Contains([]string{"base", "conditional", "optional"}, c.Applicability) { + return fmt.Errorf("invalid component coordinate or source") + } + components[c.Name] = true + } + models := map[string]bool{} + for _, h := range r.Hardware { + if h.Family == "" || len(h.Models) == 0 || h.BFB.Version == "" || !strings.HasPrefix(h.BFB.URL, "https://") || !sources[h.BFB.SourceRef] || (h.BFB.Digest == "" && (h.BFB.ReasonCode == "" || h.BFB.Message == "")) { + return fmt.Errorf("invalid hardware applicability or artifact provenance") + } + for _, model := range h.Models { + model = strings.ToLower(strings.TrimSpace(model)) + if model == "" || models[model] { + return fmt.Errorf("missing or ambiguous hardware model") + } + models[model] = true + } + seen := map[string]bool{} + for _, req := range h.Requirements { + if req.Component == "" || req.Scope == "" || seen[req.Component+"/"+req.Scope] || !sources[req.SourceRef] { + return fmt.Errorf("invalid hardware requirement or source") + } + seen[req.Component+"/"+req.Scope] = true + switch req.State { + case "known": + if req.Minimum == "" && req.Target == "" { + return fmt.Errorf("known requirement has no minimum or target") + } + case "unknown": + if req.Minimum != "" || req.Target != "" || req.ReasonCode == "" || req.Message == "" { + return fmt.Errorf("unknown requirement needs a reason and no inferred version") + } + default: + return fmt.Errorf("invalid requirement state") + } + } + } + return nil +} diff --git a/pkg/dpf/releases/releases.yaml b/pkg/dpf/releases/releases.yaml new file mode 100644 index 00000000..fe2d5950 --- /dev/null +++ b/pkg/dpf/releases/releases.yaml @@ -0,0 +1,108 @@ +# Copyright 2026 NVIDIA CORPORATION & AFFILIATES +# SPDX-License-Identifier: Apache-2.0 +# Curated software coordinates, not a Launch Kit hardware qualification catalog. +releases: + "26.4.0": + version: "26.4.0" + operator: + chart: dpf-operator + chartRepository: https://helm.ngc.nvidia.com/nvidia/doca + version: v26.4.0 + chartDigest: sha256:0f9a18a0d64958e2f1769bdb468a46ad65e0ea485dadb86cad9e6457b3cec69a + image: nvcr.io/nvidia/doca/dpf-system:v26.4.0 + sourceRef: published-chart + components: + - {name: argocd, version: v3.3.0, repository: quay.io/argoproj/argocd, applicability: base, sourceRef: platform-support} + - {name: cert-manager-controller, version: v1.19.3, repository: quay.io/jetstack/cert-manager-controller, applicability: base, sourceRef: platform-support} + - {name: flannel, version: v0.26.5, repository: docker.io/flannel/flannel, applicability: base, sourceRef: platform-support} + - {name: multus, version: v3.9.3, repository: ghcr.io/k8snetworkplumbingwg/multus-cni, applicability: base, sourceRef: platform-support} + - {name: nvidia-k8s-ipam, version: v0.3.5, repository: ghcr.io/mellanox/nvidia-k8s-ipam, applicability: base, sourceRef: platform-support} + - {name: maintenance-operator, version: v0.3.0, repository: ghcr.io/mellanox/maintenance-operator, applicability: base, sourceRef: platform-support} + - {name: network-operator, version: v26.1.0, repository: nvcr.io/nvidia/cloud-native/network-operator, applicability: base, sourceRef: platform-support} + - {name: node-feature-discovery, version: v0.18.3, repository: registry.k8s.io/nfd/node-feature-discovery, applicability: base, sourceRef: platform-support} + - {name: sriov-device-plugin, version: v3.11.0, repository: ghcr.io/k8snetworkplumbingwg/sriov-network-device-plugin, applicability: base, sourceRef: platform-support} + - {name: etcd-defrag, version: v0.22.0, repository: ghcr.io/ahrtr/etcd-defrag, applicability: optional, sourceRef: platform-support} + - {name: kamaji, version: v1.34.0-25.9.3, repository: ghcr.io/nvidia/kamaji, applicability: conditional, sourceRef: platform-support} + - {name: local-path-provisioner, version: v0.0.34, repository: docker.io/rancher/local-path-provisioner, applicability: conditional, sourceRef: platform-support} + - {name: kube-state-metrics, version: v2.18.0, repository: registry.k8s.io/kube-state-metrics/kube-state-metrics, applicability: optional, sourceRef: platform-support} + - {name: node-problem-detector, version: v1.35.1, repository: registry.k8s.io/node-problem-detector/node-problem-detector, applicability: optional, sourceRef: platform-support} + - {name: prometheus, version: v2.54.1, repository: quay.io/prometheus/prometheus, applicability: optional, sourceRef: platform-support} + - {name: grafana, version: "11.1.0", repository: docker.io/grafana/grafana, applicability: optional, sourceRef: platform-support} + hardware: + - family: BlueField-3 + models: [B3210, B3220, B3240] + bfb: + version: "3.4.0-92" + url: https://content.mellanox.com/BlueField/BFBs/Ubuntu24.04/bf-bundle-3.4.0-92_26.04_ubuntu-24.04_64k_prod.bfb + reasonCode: ARTIFACT_DIGEST_NOT_RECORDED + message: The upstream release guide supplies a versioned BFB URL but no digest; immutable artifact verification remains required before provisioning. + sourceRef: zero-trust-env + requirements: + - component: nic + scope: dpu + state: known + minimum: "32.38.1002" + sourceRef: platform-support + - component: mft + scope: host + state: known + minimum: "4.33.0-169" + sourceRef: platform-support + - component: bmc + scope: dpu + state: unknown + reasonCode: RELEASE_REQUIREMENT_NOT_DOCUMENTED + message: The pinned platform requirements do not specify a BMC firmware minimum or target. The PoC capability threshold is not substituted for an official release requirement. + sourceRef: platform-support + - component: cec + scope: dpu + state: unknown + reasonCode: RELEASE_REQUIREMENT_NOT_DOCUMENTED + message: No standalone CEC requirement is declared by the pinned platform requirements; retain the observed version without inferring compatibility. + sourceRef: platform-support + - component: atf + scope: dpu + state: unknown + reasonCode: RELEASE_REQUIREMENT_NOT_DOCUMENTED + message: No standalone ATF requirement is declared by the pinned platform requirements; retain the observed version without inferring compatibility. + sourceRef: platform-support + - component: uefi + scope: dpu + state: unknown + reasonCode: RELEASE_REQUIREMENT_NOT_DOCUMENTED + message: No standalone UEFI requirement is declared by the pinned platform requirements; retain the observed version without inferring compatibility. + sourceRef: platform-support + - component: dpu-bsp + scope: dpu + state: unknown + reasonCode: RELEASE_REQUIREMENT_NOT_DOCUMENTED + message: The guide selects a BFB bundle but does not specify the BSP inventory version string; do not equate a bundle version with a BSP version. + sourceRef: zero-trust-env + - component: dpu-os + scope: dpu + state: unknown + reasonCode: RELEASE_REQUIREMENT_NOT_DOCUMENTED + message: The guide selects a BFB bundle but does not specify the DPU OS inventory version string; do not equate a bundle version with an OS version. + sourceRef: zero-trust-env + - component: golden + scope: dpu + state: unknown + reasonCode: RELEASE_REQUIREMENT_NOT_DOCUMENTED + message: No standalone golden-image requirement is declared by the pinned platform requirements; retain the observed version without inferring compatibility. + sourceRef: platform-support + sources: + - id: platform-support + url: https://github.com/NVIDIA/doca-platform/blob/8551df0d3ac4ed51a67d4a5148a7646493a30e10/docs/public/platform-support.md + revision: 8551df0d3ac4ed51a67d4a5148a7646493a30e10 + - id: zero-trust-env + url: https://github.com/NVIDIA/doca-platform/blob/8551df0d3ac4ed51a67d4a5148a7646493a30e10/docs/public/user-guides/zero-trust/use-cases/passthrough/manifests/00-env-vars/envvars.env + revision: 8551df0d3ac4ed51a67d4a5148a7646493a30e10 + - id: published-chart + url: https://helm.ngc.nvidia.com/nvidia/doca/charts/dpf-operator-v26.4.0.tgz + revision: sha256:0f9a18a0d64958e2f1769bdb468a46ad65e0ea485dadb86cad9e6457b3cec69a + limitations: + - Catalog applicability is limited to the exact listed BlueField-3 models; BlueField-4 and generic family-only observations require a separately reviewed hardware BOM. + - Software coordinates are upstream release metadata, not proof of installed versions, active firmware, runtime compatibility, or Launch Kit hardware qualification. + - Component applicability mirrors the upstream component matrix; it does not select services or authorize changing the independent host cluster. + - The published chart records build commit 50fab50af0bfa67be22e8bb4768548ad7dae57bf, separately from the public source tag commit used for documentation. + - BFB and component image bytes are not digest-locked by this catalog; provisioning must resolve and verify immutable artifacts before use. diff --git a/pkg/dpf/releases/releases_test.go b/pkg/dpf/releases/releases_test.go new file mode 100644 index 00000000..3b3d9196 --- /dev/null +++ b/pkg/dpf/releases/releases_test.go @@ -0,0 +1,104 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package releases + +import ( + "strings" + "sync" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestReleases(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "DPF release catalog") } + +var _ = Describe("release BOM resolution", func() { + It("defaults to a pinned full release and retains separately sourced coordinates", func() { + r, err := Resolve("") + Expect(err).NotTo(HaveOccurred()) + Expect(r.Version).To(Equal(DefaultRelease)) + Expect(r.Operator.Version).To(Equal("v26.4.0")) + Expect(r.Operator.Image).To(Equal("nvcr.io/nvidia/doca/dpf-system:v26.4.0")) + Expect(r.Operator.ChartDigest).To(Equal("sha256:0f9a18a0d64958e2f1769bdb468a46ad65e0ea485dadb86cad9e6457b3cec69a")) + Expect(SupportedReleases()).To(Equal([]string{"26.4.0"})) + }) + It("rejects unreviewed releases and does not interpret latest, prefixes or whitespace", func() { + for _, version := range []string{"latest", "26.4", "26.4.1", "v26.4.0", " 26.4.0", "26.4.0 "} { + _, err := Resolve(version) + Expect(err).To(MatchError(ContainSubstring("unknown DPF release"))) + } + }) + It("keeps unknown BMC requirements distinct from a documented NIC minimum", func() { + r, err := Resolve(DefaultRelease) + Expect(err).NotTo(HaveOccurred()) + h, ok := r.RequirementsFor("b3240") + Expect(ok).To(BeTrue()) + Expect(h.Family).To(Equal("BlueField-3")) + Expect(h.Requirements).To(ContainElement(And(HaveField("Component", "nic"), HaveField("Minimum", "32.38.1002"), HaveField("State", "known")))) + Expect(h.Requirements).To(ContainElement(And(HaveField("Component", "bmc"), HaveField("Minimum", ""), HaveField("Target", ""), HaveField("State", "unknown"), HaveField("ReasonCode", "RELEASE_REQUIREMENT_NOT_DOCUMENTED")))) + Expect(h.BFB.Digest).To(BeEmpty()) + Expect(h.BFB.ReasonCode).To(Equal("ARTIFACT_DIGEST_NOT_RECORDED")) + }) + It("does not apply BF3 requirements to BF4, family names, or partial model matches", func() { + r, err := Resolve(DefaultRelease) + Expect(err).NotTo(HaveOccurred()) + for _, model := range []string{"", "BlueField-3", "BlueField-4", "B4240", "B3240 extra", "XXB3240"} { + _, ok := r.RequirementsFor(model) + Expect(ok).To(BeFalse()) + } + var missing *Release + _, ok := missing.RequirementsFor("B3240") + Expect(ok).To(BeFalse()) + }) + It("isolates returned nested values from the catalog and hardware lookup", func() { + r, err := Resolve(DefaultRelease) + Expect(err).NotTo(HaveOccurred()) + h, ok := r.RequirementsFor("B3240") + Expect(ok).To(BeTrue()) + h.Models[0] = "altered" + h.Requirements[0].Minimum = "0" + Expect(r.Hardware[0].Models[0]).To(Equal("B3210")) + Expect(r.Hardware[0].Requirements[0].Minimum).To(Equal("32.38.1002")) + r.Hardware[0].Models[0] = "changed" + r.Hardware[0].Requirements[0].Minimum = "0" + r.Components[0].Version = "changed" + r.Sources[0].URL = "changed" + r.Limitations[0] = "changed" + next, err := Resolve(DefaultRelease) + Expect(err).NotTo(HaveOccurred()) + Expect(next.Hardware[0].Models[0]).To(Equal("B3210")) + Expect(next.Hardware[0].Requirements[0].Minimum).To(Equal("32.38.1002")) + Expect(next.Components[0].Version).To(Equal("v3.3.0")) + Expect(next.Sources[0].URL).To(HavePrefix("https://github.com/")) + Expect(next.Limitations[0]).NotTo(Equal("changed")) + }) + It("allows concurrent independent resolutions", func() { + var wg sync.WaitGroup + for range 10 { + wg.Add(1) + go func() { + defer GinkgoRecover() + defer wg.Done() + r, err := Resolve(DefaultRelease) + Expect(err).NotTo(HaveOccurred()) + r.Hardware[0].Models[0] = "independent" + }() + } + wg.Wait() + }) + It("rejects malformed catalogs rather than hiding missing provenance", func() { + for _, data := range []string{ + "releases: {}", + string(catalogYAML) + "\n---\nreleases: {}\n", + strings.Replace(string(catalogYAML), "sourceRef: published-chart", "sourceRef: absent", 1), + strings.Replace(string(catalogYAML), "state: unknown", "state: known", 1), + strings.Replace(string(catalogYAML), "models: [B3210, B3220, B3240]", "models: [B3210, B3210]", 1), + strings.Replace(string(catalogYAML), "version: \"26.4.0\"", "version: \"26.8.0\"", 1), + strings.Replace(string(catalogYAML), "chart: dpf-operator", "unknown: true", 1), + } { + _, err := load([]byte(data)) + Expect(err).To(HaveOccurred()) + } + }) +}) diff --git a/pkg/dpf/releases/summary.go b/pkg/dpf/releases/summary.go new file mode 100644 index 00000000..54bf15ae --- /dev/null +++ b/pkg/dpf/releases/summary.go @@ -0,0 +1,188 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package releases + +import ( + "encoding/hex" + "fmt" + "net/url" + "regexp" + "slices" + "strings" + "unicode" +) + +// Summary is the persisted software BOM. Catalog research provenance remains in +// Release; omitted minima mean unknown requirements, never inferred constraints. +type Summary struct { + Version string `json:"version" yaml:"version"` + Operator OperatorSummary `json:"operator" yaml:"operator"` + Components []ComponentSummary `json:"components" yaml:"components"` + Hardware []HardwareSummary `json:"hardware" yaml:"hardware"` +} + +type OperatorSummary struct { + Chart string `json:"chart" yaml:"chart"` + ChartRepository string `json:"chartRepository" yaml:"chartRepository"` + Version string `json:"version" yaml:"version"` + ChartDigest string `json:"chartDigest" yaml:"chartDigest"` + Image string `json:"image" yaml:"image"` +} + +type ComponentSummary struct { + Name string `json:"name" yaml:"name"` + Version string `json:"version" yaml:"version"` + Repository string `json:"repository" yaml:"repository"` + Applicability string `json:"applicability" yaml:"applicability"` +} + +type ArtifactSummary struct { + Version string `json:"version" yaml:"version"` + URL string `json:"url" yaml:"url"` + Digest string `json:"digest,omitempty" yaml:"digest,omitempty"` +} + +type HardwareSummary struct { + Family string `json:"family" yaml:"family"` + Models []string `json:"models" yaml:"models"` + BFB ArtifactSummary `json:"bfb" yaml:"bfb"` + MinimumNICFirmware string `json:"minimumNICFirmware,omitempty" yaml:"minimumNICFirmware,omitempty"` + MinimumHostMFT string `json:"minimumHostMFT,omitempty" yaml:"minimumHostMFT,omitempty"` + MinimumBMCFirmware string `json:"minimumBMCFirmware,omitempty" yaml:"minimumBMCFirmware,omitempty"` + MinimumCECFirmware string `json:"minimumCECFirmware,omitempty" yaml:"minimumCECFirmware,omitempty"` + MinimumATFFirmware string `json:"minimumATFFirmware,omitempty" yaml:"minimumATFFirmware,omitempty"` + MinimumUEFIFirmware string `json:"minimumUEFIFirmware,omitempty" yaml:"minimumUEFIFirmware,omitempty"` + MinimumDPUBSP string `json:"minimumDPUBSP,omitempty" yaml:"minimumDPUBSP,omitempty"` + MinimumDPUOS string `json:"minimumDPUOS,omitempty" yaml:"minimumDPUOS,omitempty"` + MinimumGoldenImage string `json:"minimumGoldenImage,omitempty" yaml:"minimumGoldenImage,omitempty"` +} + +// minimumFields fixes the supported requirement names and measurement scopes. +// It does not consult a newer catalog when decoding a saved inventory. +func (h *HardwareSummary) minimumFields() []struct { + component, scope string + value *string +} { + return []struct { + component, scope string + value *string + }{ + {"nic", "dpu", &h.MinimumNICFirmware}, + {"mft", "host", &h.MinimumHostMFT}, + {"bmc", "dpu", &h.MinimumBMCFirmware}, + {"cec", "dpu", &h.MinimumCECFirmware}, + {"atf", "dpu", &h.MinimumATFFirmware}, + {"uefi", "dpu", &h.MinimumUEFIFirmware}, + {"dpu-bsp", "dpu", &h.MinimumDPUBSP}, + {"dpu-os", "dpu", &h.MinimumDPUOS}, + {"golden", "dpu", &h.MinimumGoldenImage}, + } +} + +// Summarize projects a curated release into facts needed by inventory consumers. +// It returns independent slices and never adds a digest or a numeric minimum. +func Summarize(r Release) Summary { + s := Summary{ + Version: r.Version, + Operator: OperatorSummary{Chart: r.Operator.Chart, ChartRepository: r.Operator.ChartRepository, + Version: r.Operator.Version, ChartDigest: r.Operator.ChartDigest, Image: r.Operator.Image}, + Components: make([]ComponentSummary, 0, len(r.Components)), + Hardware: make([]HardwareSummary, 0, len(r.Hardware)), + } + for _, c := range r.Components { + s.Components = append(s.Components, ComponentSummary{Name: c.Name, Version: c.Version, Repository: c.Repository, Applicability: c.Applicability}) + } + for _, h := range r.Hardware { + out := HardwareSummary{Family: h.Family, Models: slices.Clone(h.Models), + BFB: ArtifactSummary{Version: h.BFB.Version, URL: h.BFB.URL, Digest: h.BFB.Digest}} + for _, field := range out.minimumFields() { + for _, requirement := range h.Requirements { + if requirement.Component == field.component && requirement.Scope == field.scope && requirement.State == "known" { + *field.value = requirement.Minimum + } + } + } + s.Hardware = append(s.Hardware, out) + } + return s +} + +var ( + summaryReleaseVersion = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+$`) + summaryMinimumVersion = regexp.MustCompile(`^[0-9]+([.-][0-9]+)*$`) +) + +// Release restores consumer semantics from saved facts, without resolving the +// current catalog. Unknown minima stay unknown; BFB targets are never minima. +// The caller must use strict YAML/JSON decoding to reject unknown fields. +func (s Summary) Release() (Release, error) { + if !summaryReleaseVersion.MatchString(s.Version) || s.Operator.Version != "v"+s.Version || + !summaryToken(s.Operator.Chart) || !summaryHTTPS(s.Operator.ChartRepository) || + !summaryDigest(s.Operator.ChartDigest) || !summaryToken(s.Operator.Image) { + return Release{}, fmt.Errorf("invalid saved DPF release coordinates") + } + if len(s.Components) == 0 || len(s.Hardware) == 0 { + return Release{}, fmt.Errorf("saved DPF software BOM requires components and hardware applicability") + } + r := Release{Version: s.Version, + Operator: Operator{Chart: s.Operator.Chart, ChartRepository: s.Operator.ChartRepository, + Version: s.Operator.Version, ChartDigest: s.Operator.ChartDigest, Image: s.Operator.Image}, + Components: make([]Component, 0, len(s.Components)), Hardware: make([]HardwareBOM, 0, len(s.Hardware))} + components := map[string]bool{} + for _, c := range s.Components { + if !summaryToken(c.Name) || components[c.Name] || !summaryToken(c.Version) || !summaryToken(c.Repository) || + !slices.Contains([]string{"base", "conditional", "optional"}, c.Applicability) { + return Release{}, fmt.Errorf("invalid or duplicate saved DPF component %q", c.Name) + } + components[c.Name] = true + r.Components = append(r.Components, Component{Name: c.Name, Version: c.Version, Repository: c.Repository, Applicability: c.Applicability}) + } + models := map[string]bool{} + for _, h := range s.Hardware { + if !summaryToken(h.Family) || len(h.Models) == 0 || !summaryToken(h.BFB.Version) || !summaryHTTPS(h.BFB.URL) || + h.BFB.Digest != "" && !summaryDigest(h.BFB.Digest) { + return Release{}, fmt.Errorf("invalid saved DPF hardware applicability or BFB coordinate") + } + for _, model := range h.Models { + key := strings.ToLower(model) + if !summaryToken(model) || models[key] { + return Release{}, fmt.Errorf("invalid or duplicate saved DPF hardware model %q", model) + } + models[key] = true + } + out := HardwareBOM{Family: h.Family, Models: slices.Clone(h.Models), + BFB: Artifact{Version: h.BFB.Version, URL: h.BFB.URL, Digest: h.BFB.Digest}, Requirements: []Requirement{}} + for _, field := range h.minimumFields() { + requirement := Requirement{Component: field.component, Scope: field.scope, State: "unknown", + ReasonCode: "RELEASE_REQUIREMENT_NOT_DOCUMENTED", Message: "The saved release BOM records no minimum for this component."} + if *field.value != "" { + if !summaryMinimumVersion.MatchString(*field.value) { + return Release{}, fmt.Errorf("invalid saved DPF %s minimum", field.component) + } + requirement.State, requirement.Minimum = "known", *field.value + requirement.ReasonCode, requirement.Message = "", "" + } + out.Requirements = append(out.Requirements, requirement) + } + r.Hardware = append(r.Hardware, out) + } + return r, nil +} + +func summaryToken(s string) bool { + return s != "" && strings.IndexFunc(s, unicode.IsSpace) < 0 && strings.IndexFunc(s, unicode.IsControl) < 0 +} + +func summaryHTTPS(s string) bool { + u, err := url.Parse(s) + return err == nil && summaryToken(s) && u.Scheme == "https" && u.Hostname() != "" && u.User == nil && u.RawQuery == "" && u.Fragment == "" +} + +func summaryDigest(s string) bool { + if !strings.HasPrefix(s, "sha256:") || s != strings.ToLower(s) { + return false + } + decoded, err := hex.DecodeString(strings.TrimPrefix(s, "sha256:")) + return err == nil && len(decoded) == 32 +} diff --git a/pkg/dpf/releases/summary_test.go b/pkg/dpf/releases/summary_test.go new file mode 100644 index 00000000..408ad8a2 --- /dev/null +++ b/pkg/dpf/releases/summary_test.go @@ -0,0 +1,143 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package releases + +import ( + "encoding/json" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" +) + +var _ = Describe("persisted software BOM", func() { + It("publishes artifact coordinates and named known minima without catalog research metadata", func() { + s := summaryFixture() + Expect(s.Hardware[0].MinimumNICFirmware).To(Equal("32.38.1002")) + Expect(s.Hardware[0].MinimumHostMFT).To(Equal("4.33.0-169")) + Expect(s.Hardware[0].MinimumBMCFirmware).To(BeEmpty()) + Expect(s.Hardware[0].BFB.Digest).To(BeEmpty()) + for _, encode := range []func(any) ([]byte, error){json.Marshal, yaml.Marshal} { + encoded, err := encode(s) + Expect(err).NotTo(HaveOccurred()) + for _, removed := range []string{"sourceRef", "sources", "revision", "requirements", "limitations", "ARTIFACT_DIGEST_NOT_RECORDED", "RELEASE_REQUIREMENT_NOT_DOCUMENTED"} { + Expect(string(encoded)).NotTo(ContainSubstring(removed)) + } + Expect(string(encoded)).To(ContainSubstring("minimumNICFirmware")) + Expect(string(encoded)).To(ContainSubstring("minimumHostMFT")) + } + }) + + It("preserves known and unknown requirement meanings across a serialized round trip", func() { + original, err := Resolve(DefaultRelease) + Expect(err).NotTo(HaveOccurred()) + encoded, err := yaml.Marshal(Summarize(original)) + Expect(err).NotTo(HaveOccurred()) + var saved Summary + Expect(yaml.Unmarshal(encoded, &saved)).To(Succeed()) + restored, err := saved.Release() + Expect(err).NotTo(HaveOccurred()) + Expect(restored.Hardware[0].Requirements).To(HaveLen(len(original.Hardware[0].Requirements))) + for i, requirement := range original.Hardware[0].Requirements { + actual := restored.Hardware[0].Requirements[i] + Expect(actual.Component).To(Equal(requirement.Component)) + Expect(actual.Scope).To(Equal(requirement.Scope)) + Expect(actual.State).To(Equal(requirement.State)) + Expect(actual.Minimum).To(Equal(requirement.Minimum)) + Expect(actual.Target).To(Equal(requirement.Target)) + Expect(actual.ReasonCode).To(Equal(requirement.ReasonCode)) + if requirement.State == "unknown" { + Expect(actual.Message).NotTo(BeEmpty()) + } + } + Expect(restored.Hardware[0].BFB.Digest).To(BeEmpty()) + Expect(restored.Hardware[0].BFB.Version).To(Equal(original.Hardware[0].BFB.Version)) + Expect(restored.Hardware[0].BFB.URL).To(Equal(original.Hardware[0].BFB.URL)) + for _, model := range []string{"B4240", "BlueField-3", "BlueField-4", "B3240 extra"} { + _, supported := restored.RequirementsFor(model) + Expect(supported).To(BeFalse()) + } + Expect(restored.Sources).To(BeEmpty()) + Expect(restored.Limitations).To(BeEmpty()) + }) + + It("does not infer omitted minima or replace stored coordinates with catalog defaults", func() { + s := summaryFixture() + s.Version, s.Operator.Version = "99.1.2", "v99.1.2" + s.Operator.Image = "registry.example/operator:stored" + s.Components[0].Version = "v0.0.1-stored" + s.Hardware[0].BFB.URL = "https://artifacts.example/stored.bfb" + s.Hardware[0].BFB.Digest = "sha256:" + strings.Repeat("a", 64) + s.Hardware[0].MinimumNICFirmware, s.Hardware[0].MinimumHostMFT = "", "" + r, err := s.Release() + Expect(err).NotTo(HaveOccurred()) + Expect(r.Version).To(Equal(s.Version)) + Expect(r.Operator.Image).To(Equal(s.Operator.Image)) + Expect(r.Components[0].Version).To(Equal(s.Components[0].Version)) + Expect(r.Hardware[0].BFB.URL).To(Equal(s.Hardware[0].BFB.URL)) + Expect(r.Hardware[0].BFB.Digest).To(Equal(s.Hardware[0].BFB.Digest)) + for _, requirement := range r.Hardware[0].Requirements { + Expect(requirement.State).To(Equal("unknown")) + Expect(requirement.Minimum).To(BeEmpty()) + Expect(requirement.Target).To(BeEmpty()) + } + }) + + It("isolates mutable values from both the catalog and decoded summary", func() { + r, err := Resolve(DefaultRelease) + Expect(err).NotTo(HaveOccurred()) + s := Summarize(r) + s.Components[0].Version = "changed" + s.Hardware[0].Models[0] = "changed" + Expect(r.Components[0].Version).NotTo(Equal("changed")) + Expect(r.Hardware[0].Models[0]).NotTo(Equal("changed")) + + s = summaryFixture() + restored, err := s.Release() + Expect(err).NotTo(HaveOccurred()) + restored.Components[0].Version = "changed" + restored.Hardware[0].Models[0] = "changed" + Expect(s.Components[0].Version).NotTo(Equal("changed")) + Expect(s.Hardware[0].Models[0]).NotTo(Equal("changed")) + }) + + It("rejects malformed or ambiguous saved coordinates and constraints", func() { + for _, tc := range []struct { + name string + edit func(*Summary) + }{ + {"missing version", func(s *Summary) { s.Version = "" }}, + {"inconsistent operator version", func(s *Summary) { s.Operator.Version = "v0.0.1" }}, + {"credential-bearing URL", func(s *Summary) { s.Operator.ChartRepository = "https://user:pass@example.com" }}, + {"query URL", func(s *Summary) { s.Hardware[0].BFB.URL += "?token=private" }}, + {"insecure URL", func(s *Summary) { s.Hardware[0].BFB.URL = "http://example.com/a.bfb" }}, + {"invalid chart digest", func(s *Summary) { s.Operator.ChartDigest = "sha256:invalid" }}, + {"invalid BFB digest", func(s *Summary) { s.Hardware[0].BFB.Digest = "sha256:invalid" }}, + {"component absent", func(s *Summary) { s.Components = nil }}, + {"component duplicated", func(s *Summary) { s.Components = append(s.Components, s.Components[0]) }}, + {"component applicability", func(s *Summary) { s.Components[0].Applicability = "qualified" }}, + {"component empty version", func(s *Summary) { s.Components[0].Version = "" }}, + {"hardware absent", func(s *Summary) { s.Hardware = nil }}, + {"model absent", func(s *Summary) { s.Hardware[0].Models = nil }}, + {"model whitespace", func(s *Summary) { s.Hardware[0].Models[0] = " B3210" }}, + {"model duplicated case", func(s *Summary) { s.Hardware[0].Models = append(s.Hardware[0].Models, "b3210") }}, + {"model ambiguous hardware", func(s *Summary) { s.Hardware = append(s.Hardware, s.Hardware[0]) }}, + {"non-numeric minimum", func(s *Summary) { s.Hardware[0].MinimumNICFirmware = "current" }}, + {"unknown minimum placeholder", func(s *Summary) { s.Hardware[0].MinimumHostMFT = "unknown" }}, + } { + By(tc.name) + s := summaryFixture() + tc.edit(&s) + _, err := s.Release() + Expect(err).To(HaveOccurred()) + } + }) +}) + +func summaryFixture() Summary { + r, err := Resolve(DefaultRelease) + ExpectWithOffset(1, err).NotTo(HaveOccurred()) + return Summarize(r) +} diff --git a/pkg/dpf/sshprobe/collect.go b/pkg/dpf/sshprobe/collect.go new file mode 100644 index 00000000..ed29982b --- /dev/null +++ b/pkg/dpf/sshprobe/collect.go @@ -0,0 +1,428 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "context" + "errors" + "net" + "path" + "strconv" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +type Collector struct{} + +type runner interface { + Run(context.Context, string, string) (string, error) +} + +func (Collector) Collect(ctx context.Context, target config.Target, timeout time.Duration) (device inventory.Device, evidence map[string]inventory.Evidence, err error) { + started := time.Now() + label := target.DisplayName() + logger := log.FromContext(ctx).WithName("ssh").WithValues("endpoint", label, "role", target.Role) + ctx = log.IntoContext(ctx, logger) + output := ui.FromContext(ctx) + output.Info("[RUNNING] %s: SSH connection/session", label) + connected := false + defer func() { + if !connected { + output.Info("[FAIL] %s: SSH connection/session (%s)", label, Code(err)) + } + logger.V(1).Info("SSH collection completed", "identityState", device.ObservationState, "identityEstablished", device.ID != "", "reasonCode", device.ReasonCode, "duration", time.Since(started)) + }() + device = emptyDevice(target) + evidence = map[string]inventory.Evidence{} + if target.SSH == nil || (target.Role != "hosts" && target.Role != "dpus") { + device.ReasonCode = "SSH_CONFIGURATION_INVALID" + device.Message = "SSH discovery requires an explicit host or DPU SSH target." + return device, evidence, &Error{Code: device.ReasonCode, Stage: "target"} + } + user, password := target.SSH.Credentials() + auth := credentials{user: user, password: password, key: target.SSH.PrivateKey(), knownHosts: target.SSH.KnownHosts()} + remote, err := newReconnectingRunner(ctx, target.Endpoint, auth, timeout) + clear(auth.key) + clear(auth.knownHosts) + auth.password = "" + if err != nil { + device.ReasonCode = Code(err) + device.Message = Reason(err) + return device, evidence, err + } + connected = true + output.Info("[PASS] %s: SSH connection/session established", label) + defer func() { + output.Info("[RUNNING] %s: SSH connection cleanup", label) + cleanupStarted := time.Now() + state := "complete" + if closeErr := remote.Close(); closeErr != nil { + state = "unconfirmed" + output.Info("[UNAVAILABLE] %s: SSH connection cleanup could not be confirmed", label) + } else { + output.Info("[PASS] %s: SSH connection cleanup completed", label) + } + logger.V(1).Info("SSH connection cleanup completed", "state", state, "duration", time.Since(cleanupStarted)) + }() + return collect(ctx, target, remote) +} + +func emptyDevice(target config.Target) inventory.Device { + return inventory.Device{ + RecordKey: inventory.Digest("dpf-access-v1", []string{target.Role, target.Endpoint, target.SystemURI}), + InputRef: target.InputRef, SSHEndpoint: target.Endpoint, ObservationState: "unavailable", + Qualification: "unqualified", InventoryCompleteness: "unknown", BMCAddressState: "not-evaluated", + Authentication: inventory.Authentication{Method: "ssh", PasswordRead: "not-evaluated", ClientCertificateSupport: "unknown", DPFClientAuthentication: "not-evaluated"}, + Firmware: []inventory.Firmware{}, EvidenceRefs: []string{}, + Time: inventory.TimeObservation{State: "unknown", ReasonCode: "NOT_EVALUATED", Message: "BMC clock and NTP are not queried through OS SSH."}, + Runtime: &inventory.RuntimeObservation{Observations: map[string]inventory.Observation{}}, + } +} + +func collect(ctx context.Context, target config.Target, remote runner) (device inventory.Device, evidence map[string]inventory.Evidence, err error) { + label := target.DisplayName() + output := ui.FromContext(ctx) + stage := "Physical inventory" + output.Info("[RUNNING] %s: %s", label, stage) + device = emptyDevice(target) + evidence = map[string]inventory.Evidence{} + defer func() { + if err != nil { + output.Info("[FAIL] %s: %s (%s)", label, stage, Code(err)) + device.ReasonCode = Code(err) + device.Message = Reason(err) + var diagnostic *Error + if errors.As(err, &diagnostic) { + if _, exists := device.Runtime.Observations[diagnostic.Stage]; !exists { + device.Runtime.Observations[diagnostic.Stage] = unavailable(err, diagnostic.Tool) + } + } + } + }() + if target.Role == "dpus" { + output.Info("[UNAVAILABLE] %s: DPU BMC administrator privileges require an explicit Redfish endpoint", label) + } + device.ReaderAdapter = "linux-host-ssh-v1" + if target.Role == "dpus" { + device.ReaderAdapter = "bluefield-arm-ssh-v1" + } + privilege := "" + if target.SSH != nil && target.SSH.Sudo { + privilege = "sudo -n " + } + add := func(stage, property string) string { + id := inventory.Digest("dpf-evidence-v1", []string{device.RecordKey, stage, property}) + evidence[id] = inventory.Evidence{Kind: "ssh", InputRef: target.InputRef, ResourceURI: stage, Property: property, ObservedAt: time.Now().UTC(), Vantage: target.Role + "-os", Adapter: device.ReaderAdapter, Authenticated: true} + device.EvidenceRefs = append(device.EvidenceRefs, id) + return id + } + read := func(stage, command string) (string, error) { + text, e := remote.Run(ctx, stage, command) + if e != nil { + device.Runtime.Observations[stage] = unavailable(e, commandTool(command)) + } + return text, e + } + if target.Role == "dpus" { + dmi, e := read("arm-dmi", privilege+"dmidecode -s system-serial-number") + if e != nil { + return device, evidence, e + } + fru, e := read("arm-fru", privilege+"ipmitool fru print 0") + if e != nil { + return device, evidence, e + } + serial, model, maker, e := armIdentity(dmi, fru) + if e != nil { + return device, evidence, e + } + device.SerialNumber, device.SerialNumberRaw = serial, strings.TrimSpace(dmi) + device.Model, device.Manufacturer = model, maker + device.IdentityKey = []string{"nvidia", serial} + device.ID = inventory.PhysicalID("dpu", device.IdentityKey) + add("arm-dmi", "SerialNumber") + add("arm-fru", "Product Serial") + fruFields, _ := fields(fru) + if board := fruFields["Board Product"]; board != "" && len(board) <= 256 && !strings.ContainsAny(board, "\x00\r\n") { + device.Runtime.Observations["board-product"] = observed("ipmitool", board) + add("arm-fru", "Board Product") + } else { + device.Runtime.Observations["board-product"] = missing("ipmitool") + } + device.BMCAddressState = "unavailable" + lan, e := read("arm-ipmi-lan", privilege+"ipmitool lan print") + if e == nil { + device.BMCEndpoint, e = bmcAddress(lan) + } + if e == nil { + device.BMCAddressState = "observed-local-ipmi" + add("arm-ipmi-lan", "IP Address") + device.Runtime.Observations["bmcAddress"] = inventory.Observation{State: "observed", Values: []string{device.BMCEndpoint}} + } else { + device.Runtime.Observations["bmcAddress"] = unavailable(e, "ipmitool") + } + } else { + uuid, e := read("host-dmi", privilege+"dmidecode -s system-uuid") + if e != nil { + return device, evidence, e + } + uuid = strings.TrimSpace(uuid) + if !validUUID(uuid) { + return device, evidence, &Error{Code: "SSH_HOST_IDENTITY_INVALID", Stage: "host-dmi"} + } + device.SystemUUID = uuid + device.IdentityKey = []string{"system-uuid", strings.ToLower(uuid)} + device.ID = inventory.PhysicalID("host", device.IdentityKey) + add("host-dmi", "UUID") + pci, e := read("host-pci", "lspci -Dnn -d 15b3:") + if e != nil { + return device, evidence, e + } + cards, e := pciCards(pci) + if e != nil { + return device, evidence, e + } + pciEvidence := add("host-pci", "PCIAddress") + for _, card := range cards { + a := inventory.Adapter{Key: inventory.Digest("dpf-adapter-pci-v1", []string{device.ID, card.address[:len(card.address)-2]}), PCIAddress: card.address, VendorID: card.vendor, DeviceID: card.device, Model: card.model, IdentityState: "unknown", ReasonCode: "CARD_SERIAL_MISSING", Selection: "not-selected", EvidenceRefs: []string{pciEvidence}} + text, e := read("host-vpd:"+card.address, privilege+"lspci -D -vv -s "+card.address) + if e == nil { + serial, slot, parseErr := vpd(text) + if parseErr != nil { + return device, evidence, parseErr + } + a.SerialNumber, a.SerialNumberRaw, a.Slot = serial, serial, slot + a.EvidenceRefs = append(a.EvidenceRefs, add("host-vpd:"+card.address, "SerialNumber")) + if serial != "" { + a.IdentityState = "known" + a.ReasonCode = "" + } + } else { + a.ReasonCode = Code(e) + } + for _, bdf := range card.functions { + a.PCIFunctions = append(a.PCIFunctions, collectPF(ctx, remote, bdf, device.Runtime, add)) + } + device.Adapters = append(device.Adapters, a) + } + device.Runtime.Observations["pciScope"] = inventory.Observation{State: "observed", Values: []string{"NVIDIA network PCI functions; grouped by host-local PCI slot"}} + } + output.Info("[RUNNING] %s: Physical inventory read completed; identity recheck pending", label) + stage = "Optional OS/network, firmware and tool observations" + output.Info("[RUNNING] %s: %s", label, stage) + collectNetwork(ctx, remote, device.Runtime, add) + collectRuntime(ctx, remote, device.Runtime, add) + collectHardware(ctx, remote, &device, privilege, target.Role, add) + observed, unavailableCount := optionalCounts(device.Runtime) + if unavailableCount > 0 { + output.Info("[UNAVAILABLE] %s: Optional OS/network/firmware reads completed; %d observed, %d unavailable or unknown; inspect the detailed report for reasons, compatibility not evaluated", label, observed, unavailableCount) + } else { + output.Info("[PASS] %s: Optional OS/network/firmware reads completed; %d observations measured, compatibility not evaluated", label, observed) + } + // Recheck physical identity after optional observations, so disconnects and + // device changes cannot publish stale identity as a successful observation. + stage = "Physical identity recheck" + output.Info("[RUNNING] %s: %s", label, stage) + if target.Role == "dpus" { + dmi, e := read("arm-dmi-recheck", privilege+"dmidecode -s system-serial-number") + if e != nil { + return device, evidence, e + } + fru, e := read("arm-fru-recheck", privilege+"ipmitool fru print 0") + if e != nil { + return device, evidence, e + } + serial, model, maker, e := armIdentity(dmi, fru) + if e != nil { + return device, evidence, e + } + if serial != device.SerialNumber || model != device.Model || maker != device.Manufacturer { + return device, evidence, &Error{Code: "INVENTORY_CHANGED", Stage: "arm-identity-recheck"} + } + } else { + uuid, e := read("host-dmi-recheck", privilege+"dmidecode -s system-uuid") + if e != nil { + return device, evidence, e + } + if strings.TrimSpace(uuid) != device.SystemUUID { + return device, evidence, &Error{Code: "INVENTORY_CHANGED", Stage: "host-identity-recheck"} + } + pci, e := read("host-pci-recheck", "lspci -Dnn -d 15b3:") + if e != nil { + return device, evidence, e + } + cards, e := pciCards(pci) + if e != nil { + return device, evidence, e + } + if len(cards) != len(device.Adapters) { + return device, evidence, &Error{Code: "INVENTORY_CHANGED", Stage: "host-pci-recheck"} + } + for i, card := range cards { + a := device.Adapters[i] + if card.address != a.PCIAddress || card.vendor != a.VendorID || card.device != a.DeviceID || card.model != a.Model { + return device, evidence, &Error{Code: "INVENTORY_CHANGED", Stage: "host-pci-recheck"} + } + if len(card.functions) != len(a.PCIFunctions) { + return device, evidence, &Error{Code: "INVENTORY_CHANGED", Stage: "host-pci-functions-recheck"} + } + for j, bdf := range card.functions { + if bdf != a.PCIFunctions[j].Address { + return device, evidence, &Error{Code: "INVENTORY_CHANGED", Stage: "host-pci-functions-recheck"} + } + } + if a.SerialNumber == "" { + continue + } + text, e := read("host-vpd-recheck:"+card.address, privilege+"lspci -D -vv -s "+card.address) + if e != nil { + return device, evidence, e + } + serial, slot, e := vpd(text) + if e != nil { + return device, evidence, e + } + if serial != a.SerialNumber || slot != a.Slot { + return device, evidence, &Error{Code: "INVENTORY_CHANGED", Stage: "host-vpd-recheck"} + } + } + } + if ctx.Err() != nil { + return device, evidence, &Error{Code: "SSH_TIMEOUT_OR_CANCELLED", Stage: "collection"} + } + device.ObservationState = "observed" + output.Info("[PASS] %s: Physical identity recheck verified; connection cleanup pending", label) + device.InventoryCompleteness = "observed" + for _, a := range device.Adapters { + if a.IdentityState != "known" { + device.InventoryCompleteness = "partial" + } + } + return device, evidence, nil +} + +func collectPF(ctx context.Context, remote runner, bdf string, runtime *inventory.RuntimeObservation, add func(string, string) string) inventory.PCIFunction { + pf := inventory.PCIFunction{Address: bdf, Netdevs: []string{}} + base := "/sys/bus/pci/devices/" + bdf + for _, p := range []struct{ field, command string }{{"driver", "readlink " + base + "/driver"}, {"numa", "cat " + base + "/numa_node"}, {"netdevs", "ls -1 " + base + "/net"}} { + stage := "host-pf:" + bdf + ":" + p.field + text, err := remote.Run(ctx, stage, p.command) + if err == nil { + text = strings.TrimSpace(text) + switch p.field { + case "driver": + if text == "" || strings.ContainsAny(text, "\r\n\t ") { + err = &Error{Code: "SSH_RUNTIME_INVALID", Stage: stage} + } else { + pf.Driver = path.Base(text) + } + case "numa": + n, e := strconv.Atoi(text) + if e != nil || n < -1 { + err = &Error{Code: "SSH_RUNTIME_INVALID", Stage: stage} + } else { + pf.NUMANode = text + } + case "netdevs": + if text != "" { + pf.Netdevs = strings.Split(text, "\n") + } + } + } + if err != nil { + runtime.Observations[stage] = unavailable(err, commandTool(p.command)) + } else { + runtime.Observations[stage] = inventory.Observation{State: "observed"} + add(stage, p.field) + } + } + return pf +} + +func collectRuntime(ctx context.Context, remote runner, runtime *inventory.RuntimeObservation, add func(string, string) string) { + for _, probe := range []struct{ stage, command string }{ + {"hostname", "hostname"}, {"os", "cat /etc/os-release"}, {"interfaces", "ip -j address show"}, {"routes", "ip -j -4 route show"}, {"dns", "cat /etc/resolv.conf"}, + } { + text, err := remote.Run(ctx, probe.stage, probe.command) + if err == nil { + switch probe.stage { + case "hostname": + runtime.Hostname = strings.TrimSpace(text) + if runtime.Hostname == "" || strings.ContainsAny(runtime.Hostname, "\r\n\t ") { + runtime.Hostname = "" + err = &Error{Code: "SSH_RUNTIME_INVALID", Stage: probe.stage} + } + case "os": + for _, line := range strings.Split(text, "\n") { + if value, ok := strings.CutPrefix(line, "PRETTY_NAME="); ok { + runtime.OS = strings.Trim(value, "\"") + } + } + if runtime.OS == "" { + err = &Error{Code: "SSH_RUNTIME_INVALID", Stage: probe.stage} + } + case "interfaces": + runtime.Interfaces, err = interfaces(text) + case "routes": + runtime.Routes, err = routes(text, runtime.Interfaces) + case "dns": + for _, line := range strings.Split(text, "\n") { + f := strings.Fields(line) + if len(f) >= 2 && f[0] == "nameserver" { + address, _, _ := strings.Cut(f[1], "%") + if net.ParseIP(address) == nil { + err = &Error{Code: "SSH_RUNTIME_INVALID", Stage: probe.stage} + runtime.DNS = nil + break + } + runtime.DNS = append(runtime.DNS, f[1]) + } + if len(f) >= 2 && (f[0] == "search" || f[0] == "domain") && runtime.Network != nil { + runtime.Network.Resolver.Search = append(runtime.Network.Resolver.Search, f[1:]...) + } + } + } + } + if err != nil { + runtime.Observations[probe.stage] = unavailable(err, commandTool(probe.command)) + } else { + runtime.Observations[probe.stage] = inventory.Observation{State: "observed"} + add(probe.stage, probe.stage) + } + } +} + +// Count the typed optional facts, including nested card measurements. Successful +// commands alone do not make missing firmware/PSID/configuration observations pass. +func optionalCounts(runtime *inventory.RuntimeObservation) (observed, unavailableCount int) { + count := func(observation inventory.Observation) { + if observation.ReasonCode == "NOT_APPLICABLE" { + return + } + if observation.State == "observed" { + observed++ + } else { + unavailableCount++ + } + } + for _, observation := range runtime.Observations { + count(observation) + } + if runtime.Hardware != nil { + count(runtime.Hardware.RShim) + for _, card := range runtime.Hardware.Cards { + count(card.Firmware) + count(card.PSID) + count(card.Privilege) + for _, observation := range card.NVConfig { + count(observation) + } + } + } + return observed, unavailableCount +} diff --git a/pkg/dpf/sshprobe/collect_test.go b/pkg/dpf/sshprobe/collect_test.go new file mode 100644 index 00000000..e1bb53b5 --- /dev/null +++ b/pkg/dpf/sshprobe/collect_test.go @@ -0,0 +1,347 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "context" + "crypto/ed25519" + "crypto/rand" + "strings" + "sync" + "testing" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "golang.org/x/crypto/ssh" +) + +func TestSSHProbe(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "DPF SSH discovery") } + +const testUUID = "12345678-1234-1234-1234-123456789abc" +const testSerial = "CARD000123" +const testFRU = `FRU Device Description : Builtin FRU Device (ID 0) + Board Mfg : Nvidia + Board Product : B4240 + Board Serial : CARD000123 + Product Manufacturer : Nvidia + Product Name : BlueField-4 + Product Serial : CARD000123 +` +const testLAN = `Set in Progress : Set Complete +Auth Type Enable : Callback : + : User : + : Operator : +IP Address Source : DHCP Address +IP Address : 192.0.2.42 +MAC Address : 02:00:00:00:00:42 +` +const testPCI = `0000:21:00.0 Ethernet controller [0200]: Vendor BlueField-4 [15b3:a2df] +0000:21:00.1 Ethernet controller [0200]: Vendor BlueField-4 [15b3:a2df] +0000:41:00.0 Ethernet controller [0200]: Vendor ConnectX-9 [15b3:1025] +` + +type fakeRunner struct { + values map[string]string + errors map[string]error + calls []string +} + +func (r *fakeRunner) Run(_ context.Context, stage, command string) (string, error) { + r.calls = append(r.calls, stage+" "+command) + if err, ok := r.errors[stage]; ok { + return "", err + } + if value, ok := r.values[stage]; ok { + return value, nil + } + return "", &Error{Code: "SSH_COMMAND_FAILED", Stage: stage} +} + +func armRunner() *fakeRunner { + return &fakeRunner{values: map[string]string{ + "arm-dmi": testSerial + "\n", "arm-fru": testFRU, "arm-ipmi-lan": testLAN, + "arm-dmi-recheck": testSerial + "\n", "arm-fru-recheck": testFRU, + "hostname": "arm.example\n", "os": "PRETTY_NAME=\"Example Linux\"\n", + "interfaces": `[{"ifindex":2,"ifname":"oob_net0","mtu":1500,"addr_info":[{"local":"192.0.2.41","prefixlen":24}]}]`, + "routes": `[{"dst":"default","gateway":"192.0.2.1","dev":"oob_net0"}]`, + "dns": "nameserver 192.0.2.53\n", + }, errors: map[string]error{}, calls: []string{}} +} + +func hostRunner() *fakeRunner { + return &fakeRunner{values: map[string]string{ + "host-dmi": testUUID, "host-dmi-recheck": testUUID, + "host-pci": testPCI, "host-pci-recheck": testPCI, + "host-vpd:0000:21:00.0": "Physical Slot: 5\n [SN] Serial number: " + testSerial, + "host-vpd-recheck:0000:21:00.0": "Physical Slot: 5\n [SN] Serial number: " + testSerial, + "host-vpd:0000:41:00.0": "Capabilities: ", + }, errors: map[string]error{}, calls: []string{}} +} + +var _ = Describe("SSH inventory observations", func() { + It("reconnects after an optional command timeout and completes the required DPU identity recheck", func() { + for _, overflow := range []bool{false, true} { + server := startServer() + DeferCleanup(server.close) + var mu sync.Mutex + hostnameCalls, dmiCalls, fruCalls := 0, 0, 0 + server.mu.Lock() + server.onCommand = func(ch ssh.Channel, command string) bool { + mu.Lock() + switch command { + case "LC_ALL=C hostname": + hostnameCalls++ + if hostnameCalls == 1 { + mu.Unlock() + if overflow { + go func() { _, _ = ch.Write(make([]byte, maxOutput+1)) }() + } + time.Sleep(1300 * time.Millisecond) + return true + } + mu.Unlock() + return false + case "LC_ALL=C dmidecode -s system-serial-number": + dmiCalls++ + case "LC_ALL=C ipmitool fru print 0": + fruCalls++ + default: + mu.Unlock() + return false + } + mu.Unlock() + text := testSerial + "\n" + if command == "LC_ALL=C ipmitool fru print 0" { + text = testFRU + } + _, _ = ch.Write([]byte(text)) + _, _ = ch.SendRequest("exit-status", false, ssh.Marshal(struct{ Status uint32 }{0})) + return true + } + server.mu.Unlock() + runner, err := newReconnectingRunner(context.Background(), "ssh://"+server.listener.Addr().String(), server.auth(), time.Second) + Expect(err).NotTo(HaveOccurred()) + DeferCleanup(runner.Close) + + device, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://" + server.listener.Addr().String()}, runner) + Expect(err).NotTo(HaveOccurred()) + Expect(device.ObservationState).To(Equal("observed")) + Expect(device.ID).NotTo(BeEmpty()) + Expect(device.Runtime.Observations["hostname"].ReasonCode).To(Equal("SSH_TIMEOUT_OR_CANCELLED")) + mu.Lock() + Expect(dmiCalls).To(Equal(2)) + Expect(fruCalls).To(Equal(2)) + mu.Unlock() + server.mu.Lock() + connectionCount := len(server.connections) + server.mu.Unlock() + Expect(connectionCount).To(Equal(2)) + } + }) + It("keeps cancellation, changed identity, and reconnect trust/authentication failures fatal", func() { + for _, scenario := range []string{"cancel", "identity-change", "auth-rejected", "host-key-changed"} { + By(scenario) + server := startServer() + DeferCleanup(server.close) + ctx, cancel := context.WithCancel(context.Background()) + DeferCleanup(cancel) + var alternate ssh.Signer + if scenario == "host-key-changed" { + _, private, err := ed25519.GenerateKey(rand.Reader) + Expect(err).NotTo(HaveOccurred()) + alternate, err = ssh.NewSignerFromKey(private) + Expect(err).NotTo(HaveOccurred()) + server.mu.Lock() + server.alternateKey = alternate + server.rekeyAfter = 2 + server.mu.Unlock() + } + if scenario == "auth-rejected" { + server.mu.Lock() + server.rejectAfter = 2 + server.mu.Unlock() + } + var mu sync.Mutex + hostnameCalls, dmiCalls, fruCalls := 0, 0, 0 + server.mu.Lock() + server.onCommand = func(ch ssh.Channel, command string) bool { + mu.Lock() + switch command { + case "LC_ALL=C hostname": + hostnameCalls++ + if hostnameCalls == 1 { + mu.Unlock() + if scenario == "cancel" { + cancel() + } + time.Sleep(1300 * time.Millisecond) + return true + } + case "LC_ALL=C dmidecode -s system-serial-number": + dmiCalls++ + case "LC_ALL=C ipmitool fru print 0": + fruCalls++ + default: + mu.Unlock() + return false + } + serial := testSerial + if scenario == "identity-change" && dmiCalls > 1 { + serial = "REPLACED-CARD" + } + text := serial + "\n" + if command == "LC_ALL=C ipmitool fru print 0" { + text = strings.ReplaceAll(testFRU, testSerial, serial) + } + mu.Unlock() + _, _ = ch.Write([]byte(text)) + _, _ = ch.SendRequest("exit-status", false, ssh.Marshal(struct{ Status uint32 }{0})) + return true + } + server.mu.Unlock() + runner, err := newReconnectingRunner(context.Background(), "ssh://"+server.listener.Addr().String(), server.auth(), time.Second) + Expect(err).NotTo(HaveOccurred()) + DeferCleanup(runner.Close) + device, _, collectErr := collect(ctx, config.Target{Role: "dpus", Endpoint: "ssh://" + server.listener.Addr().String()}, runner) + cancel() + Expect(collectErr).To(HaveOccurred()) + Expect(device.ObservationState).NotTo(Equal("observed")) + if scenario == "identity-change" { + Expect(Code(collectErr)).To(Equal("INVENTORY_CHANGED")) + } + if scenario == "cancel" { + Expect(Code(collectErr)).To(Equal("SSH_TIMEOUT_OR_CANCELLED")) + server.mu.Lock() + connectionCount := len(server.connections) + server.mu.Unlock() + Expect(connectionCount).To(Equal(1)) + } + if scenario == "auth-rejected" { + Expect(Code(collectErr)).To(Equal("SSH_AUTHENTICATION_OR_HANDSHAKE_FAILED")) + } + if scenario == "host-key-changed" { + Expect(Code(collectErr)).To(Equal("SSH_HOST_KEY_REJECTED")) + } + if scenario == "auth-rejected" || scenario == "host-key-changed" { + server.mu.Lock() + connectionCount := len(server.connections) + server.mu.Unlock() + Expect(connectionCount).To(Equal(2)) + } + _ = runner.Close() + } + }) + It("corroborates ARM identity and observes a BMC address without contacting it", func() { + r := armRunner() + d, e, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22", InputRef: "/discovery/dpus/0", SSH: &config.SSHAccess{Sudo: true}}, r) + Expect(err).NotTo(HaveOccurred()) + Expect(d.SerialNumber).To(Equal(testSerial)) + Expect(d.BMCEndpoint).To(Equal("https://192.0.2.42")) + Expect(d.BMCAddressState).To(Equal("observed-local-ipmi")) + Expect(d.ObservationState).To(Equal("observed")) + Expect(d.Qualification).To(Equal("unqualified")) + Expect(d.Authentication.DPFClientAuthentication).To(Equal("not-evaluated")) + Expect(d.Runtime.Interfaces).To(HaveLen(1)) + Expect(d.Runtime.Routes[0].InterfaceIndex).To(Equal(2)) + Expect(d.Runtime.Routes[0].Destination).To(Equal("0.0.0.0/0")) + Expect(e).NotTo(BeEmpty()) + Expect(strings.Join(r.calls, "\n")).NotTo(ContainSubstring("192.0.2.42")) + Expect(r.calls).To(ContainElement("arm-fru sudo -n ipmitool fru print 0")) + }) + It("preserves ARM identity with an unavailable optional BMC address", func() { + r := armRunner() + r.errors["arm-ipmi-lan"] = &Error{Code: "SSH_COMMAND_FAILED", Stage: "arm-ipmi-lan"} + d, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22"}, r) + Expect(err).NotTo(HaveOccurred()) + Expect(d.ID).NotTo(BeEmpty()) + Expect(d.BMCEndpoint).To(BeEmpty()) + Expect(d.Runtime.Observations["bmcAddress"].State).To(Equal("unavailable")) + }) + It("rejects differing ARM identity sources without using hostname as identity", func() { + r := armRunner() + r.values["arm-dmi"] = "OTHER-CARD" + d, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22"}, r) + Expect(Code(err)).To(Equal("SSH_IDENTITY_CONFLICT")) + Expect(d.ID).To(BeEmpty()) + Expect(d.ObservationState).To(Equal("unavailable")) + }) + It("rejects a physical identity change during collection", func() { + r := armRunner() + r.values["arm-dmi-recheck"] = "NEW-CARD" + r.values["arm-fru-recheck"] = strings.ReplaceAll(testFRU, testSerial, "NEW-CARD") + d, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22"}, r) + Expect(Code(err)).To(Equal("INVENTORY_CHANGED")) + Expect(d.ObservationState).To(Equal("unavailable")) + }) + It("groups PCI functions by slot and retains cards with unknown serials", func() { + r := hostRunner() + d, _, err := collect(context.Background(), config.Target{Role: "hosts", Endpoint: "ssh://192.0.2.11:22"}, r) + Expect(err).NotTo(HaveOccurred()) + Expect(d.Adapters).To(HaveLen(2)) + Expect(d.SystemUUID).To(Equal(testUUID)) + Expect(d.Adapters[0].SerialNumber).To(Equal(testSerial)) + Expect(d.Adapters[0].Slot).To(Equal("5")) + Expect(d.Adapters[1].SerialNumber).To(BeEmpty()) + Expect(d.Adapters[0].IdentityState).To(Equal("known")) + Expect(d.Adapters[1].IdentityState).To(Equal("unknown")) + Expect(d.InventoryCompleteness).To(Equal("partial")) + Expect(d.Runtime.Observations["interfaces"].State).To(Equal("unavailable")) + }) + It("does not combine separate PCI slots with the same serial", func() { + r := hostRunner() + r.values["host-vpd:0000:41:00.0"] = "[SN] Serial number: " + testSerial + r.values["host-vpd-recheck:0000:41:00.0"] = "[SN] Serial number: " + testSerial + d, _, err := collect(context.Background(), config.Target{Role: "hosts", Endpoint: "ssh://192.0.2.11:22"}, r) + Expect(err).NotTo(HaveOccurred()) + Expect(d.Adapters).To(HaveLen(2)) + Expect(d.Adapters[0].Key).NotTo(Equal(d.Adapters[1].Key)) + }) + It("retains distinct per-function driver, NUMA and netdev observations", func() { + r := hostRunner() + for i, bdf := range []string{"0000:21:00.0", "0000:21:00.1"} { + r.values["host-pf:"+bdf+":driver"] = "../../../../bus/pci/drivers/mlx5_core\n" + r.values["host-pf:"+bdf+":numa"] = "0\n" + r.values["host-pf:"+bdf+":netdevs"] = []string{"hostpf0\n", "hostpf1\n"}[i] + } + d, _, err := collect(context.Background(), config.Target{Role: "hosts", Endpoint: "ssh://192.0.2.11:22"}, r) + Expect(err).NotTo(HaveOccurred()) + Expect(d.Adapters[0].PCIFunctions).To(HaveLen(2)) + Expect(d.Adapters[0].PCIFunctions[0].Netdevs).To(Equal([]string{"hostpf0"})) + Expect(d.Adapters[0].PCIFunctions[1].Netdevs).To(Equal([]string{"hostpf1"})) + Expect(d.Adapters[0].PCIFunctions[0].Driver).To(Equal("mlx5_core")) + Expect(d.Adapters[0].PCIFunctions[0].NUMANode).To(Equal("0")) + Expect(d.Runtime.Observations["host-pf:0000:41:00.0:driver"].State).To(Equal("unavailable")) + }) + It("rejects host identity or topology changes", func() { + for _, field := range []string{"host-dmi-recheck", "host-pci-recheck", "host-vpd-recheck:0000:21:00.0"} { + r := hostRunner() + r.values[field] = "" + d, _, err := collect(context.Background(), config.Target{Role: "hosts", Endpoint: "ssh://192.0.2.11:22"}, r) + Expect(err).To(HaveOccurred()) + Expect(d.ObservationState).To(Equal("unavailable")) + } + }) + It("rejects malformed and shell-bearing PCI output before any VPD command", func() { + r := hostRunner() + r.values["host-pci"] = "0000:21:00.0;touch /tmp/unwanted Ethernet controller [0200]: Vendor [15b3:a2df]" + _, _, err := collect(context.Background(), config.Target{Role: "hosts", Endpoint: "ssh://192.0.2.11:22"}, r) + Expect(Code(err)).To(Equal("SSH_PCI_INVENTORY_INVALID")) + Expect(strings.Join(r.calls, "\n")).NotTo(ContainSubstring("host-vpd")) + }) + It("rejects ambiguous FRUs, invalid identities and invalid BMC addresses", func() { + _, _, _, err := armIdentity(testSerial, testFRU+"Product Serial: OTHER\n") + Expect(err).To(HaveOccurred()) + for _, serial := range []string{"unknown", "000000", "\n", "line\nbreak"} { + Expect(validSerial(serial)).To(BeFalse()) + } + for _, uuid := range []string{"00000000-0000-0000-0000-000000000000", "ffffffff-ffff-ffff-ffff-ffffffffffff", "example"} { + Expect(validUUID(uuid)).To(BeFalse()) + } + for _, ip := range []string{"0.0.0.0", "127.0.0.1", "224.0.0.1", "192.0.2.1; echo bad"} { + _, err := bmcAddress(strings.ReplaceAll(testLAN, "192.0.2.42", ip)) + Expect(err).To(HaveOccurred()) + } + }) +}) diff --git a/pkg/dpf/sshprobe/diagnostics.go b/pkg/dpf/sshprobe/diagnostics.go new file mode 100644 index 00000000..9022abce --- /dev/null +++ b/pkg/dpf/sshprobe/diagnostics.go @@ -0,0 +1,111 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "errors" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "golang.org/x/crypto/ssh" +) + +// commandFailure classifies bounded output, but never returns it. Tool diagnostics +// can include paths and credentials; only static explanations reach artifacts. +func commandFailure(err error, stage, command, output string) *Error { + code := "SSH_COMMAND_FAILED" + var status *ssh.ExitError + if errors.As(err, &status) { + switch status.ExitStatus() { + case 126: + code = "TOOL_NOT_EXECUTABLE" + case 127: + code = "TOOL_NOT_AVAILABLE" + case 66: + if stage == "dnsmasq" { + code = "PROPERTY_NOT_EXPOSED" + } + case 77: + if stage == "dnsmasq" { + code = "PERMISSION_DENIED" + } + } + } + lower := strings.ToLower(output) + switch { + case strings.Contains(lower, "command not found"), strings.Contains(lower, "not found in path"): + code = "TOOL_NOT_AVAILABLE" + case strings.Contains(lower, "permission denied"), strings.Contains(lower, "access denied"), strings.Contains(lower, "operation not permitted"), strings.Contains(lower, "a password is required"), strings.Contains(lower, "not allowed to execute"): + code = "PERMISSION_DENIED" + case strings.Contains(lower, "failed to open device"), strings.Contains(lower, "cannot open device"): + code = "DEVICE_ACCESS_FAILED" + case strings.Contains(lower, "not supported"), strings.Contains(lower, "unsupported"), strings.Contains(lower, "unknown parameter"), strings.Contains(lower, "unrecognized option"): + code = "OPERATION_UNSUPPORTED" + } + tool := commandTool(command) + if strings.HasPrefix(command, "sudo -n ") && (strings.Contains(lower, "sudo: not found") || strings.Contains(lower, "sudo: command not found")) { + tool = "sudo" + } + return &Error{Code: code, Stage: stage, Tool: tool} +} + +func commandTool(command string) string { + f := strings.Fields(command) + if len(f) >= 3 && f[0] == "sudo" && f[1] == "-n" { + f = f[2:] + } + if len(f) == 0 { + return "" + } + return f[0] +} + +// Reason returns a safe explanation without library or remote error text. +func Reason(err error) string { + switch Code(err) { + case "TOOL_NOT_AVAILABLE": + return "Required tool is not available in the remote command environment." + case "TOOL_NOT_EXECUTABLE": + return "The remote tool exists but could not be executed." + case "PERMISSION_DENIED": + return "The configured account or noninteractive sudo policy does not permit this read." + case "DEVICE_ACCESS_FAILED": + return "The tool could not open the queried device; device access was not established." + case "OPERATION_UNSUPPORTED": + return "The installed tool or device does not support this read operation." + case "SSH_TIMEOUT_OR_CANCELLED": + return "The observation timed out or was cancelled." + case "SSH_OUTPUT_LIMIT": + return "The remote response exceeded the bounded output limit." + case "SSH_RUNTIME_INVALID": + return "The command returned missing, malformed or ambiguous observation data." + case "PROPERTY_NOT_EXPOSED": + return "The requested property is not exposed in the command response." + case "NOT_APPLICABLE": + return "This observation does not apply to this device or collection vantage." + case "SSH_HOST_KEY_REJECTED": + return "The remote SSH host key did not match the configured trust." + case "SSH_CONNECT_FAILED": + return "The configured SSH endpoint could not be reached." + case "SSH_AUTHENTICATION_OR_HANDSHAKE_FAILED": + return "SSH authentication or session negotiation failed." + default: + return "The requested read failed; no reliable observation was obtained." + } +} + +func unavailable(err error, tool string) inventory.Observation { + var e *Error + if errors.As(err, &e) && e.Tool != "" { + tool = e.Tool + } + return inventory.Observation{State: "unavailable", Values: []string{}, ReasonCode: Code(err), Message: Reason(err), Tool: tool} +} + +func missing(tool string) inventory.Observation { + return unavailable(&Error{Code: "PROPERTY_NOT_EXPOSED"}, tool) +} + +func observed(tool string, values ...string) inventory.Observation { + return inventory.Observation{State: "observed", Values: values, Tool: tool} +} diff --git a/pkg/dpf/sshprobe/hardware.go b/pkg/dpf/sshprobe/hardware.go new file mode 100644 index 00000000..76eeb09f --- /dev/null +++ b/pkg/dpf/sshprobe/hardware.go @@ -0,0 +1,236 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "context" + "regexp" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +var versionToken = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._()+-]{0,127}$`) +var bdfToken = regexp.MustCompile(`(?i)^[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-7]$`) + +func collectHardware(ctx context.Context, remote runner, device *inventory.Device, privilege, role string, add func(string, string) string) { + h := &inventory.HardwareObservation{Cards: []inventory.CardHardware{}, RShim: unavailable(&Error{Code: "NOT_APPLICABLE"}, "systemctl")} + device.Runtime.Hardware = h + version, versionErr := remote.Run(ctx, "mft-version", privilege+"mlxfwmanager --version") + if versionErr == nil { + matches := regexp.MustCompile(`(?i)\bmft\s+(\d+\.\d+\.\d+(?:-\d+)?)\b`).FindAllStringSubmatch(version, -1) + if len(matches) != 1 { + versionErr = &Error{Code: "SSH_RUNTIME_INVALID", Stage: "mft-version"} + } else { + device.Runtime.Observations["mft-version"] = observed("mlxfwmanager", matches[0][1]) + add("mft-version", "MFTVersion") + } + } + if versionErr != nil { + device.Runtime.Observations["mft-version"] = unavailable(versionErr, "mlxfwmanager") + } + var cards []pciCard + if role == "hosts" { + for _, a := range device.Adapters { + cards = append(cards, pciCard{address: a.PCIAddress, model: a.Model}) + } + text, err := remote.Run(ctx, "rshim-service", "systemctl show rshim --property=LoadState --property=ActiveState") + if err == nil { + values := []string{} + for _, line := range strings.Split(text, "\n") { + key, value, ok := strings.Cut(line, "=") + if ok && (key == "LoadState" || key == "ActiveState") && versionToken.MatchString(value) { + values = append(values, key+"="+value) + } + } + if len(values) != 2 { + err = &Error{Code: "SSH_RUNTIME_INVALID", Stage: "rshim-service"} + } else { + h.RShim = observed("systemctl", values...) + add("rshim-service", "service-state") + } + } + if err != nil { + h.RShim = unavailable(err, "systemctl") + } + } else { + text, err := remote.Run(ctx, "arm-pci", "lspci -Dnn -d 15b3:") + if err == nil { + cards, err = pciCards(text) + } + if err != nil { + device.Runtime.Observations["arm-pci"] = unavailable(err, "lspci") + return + } + device.Runtime.Observations["arm-pci"] = observed("lspci") + add("arm-pci", "PCIAddress") + } + for _, card := range cards { + // Never interpolate any address other than a validated collector-owned BDF. + if !bdfToken.MatchString(card.address) { + continue + } + c := inventory.CardHardware{PCIAddress: card.address, Domain: "host-pci", Model: card.model, NVConfig: map[string]inventory.Observation{}, EvidenceRefs: []string{}, Privilege: unavailable(&Error{Code: "NOT_APPLICABLE"}, "mlxprivhost")} + if role == "hosts" { + for _, a := range device.Adapters { + if a.PCIAddress == card.address { + c.SerialNumber = a.SerialNumber + } + } + } else { + c.Domain = "arm-pci" + stage := "arm-vpd:" + card.address + text, err := remote.Run(ctx, stage, privilege+"lspci -D -vv -s "+card.address) + if err == nil { + c.SerialNumber, _, err = vpd(text) + if err == nil && c.SerialNumber == "" { + err = &Error{Code: "PROPERTY_NOT_EXPOSED", Stage: stage} + } + } + if err != nil { + device.Runtime.Observations[stage] = unavailable(err, "lspci") + } else { + device.Runtime.Observations[stage] = observed("lspci") + c.EvidenceRefs = append(c.EvidenceRefs, add(stage, "SerialNumber")) + } + } + stage := "firmware:" + card.address + text, err := remote.Run(ctx, stage, privilege+"mlxfwmanager --query -d "+card.address) + if err == nil { + c.Firmware, c.PSID, err = parseFirmware(text, card.address) + } + if err != nil { + c.Firmware = unavailable(err, "mlxfwmanager") + c.PSID = unavailable(err, "mlxfwmanager") + } else { + c.EvidenceRefs = append(c.EvidenceRefs, add(stage, "firmware-psid")) + } + if strings.Contains(strings.ToLower(card.model), "bluefield") { + for _, parameter := range []string{"INTERNAL_CPU_OFFLOAD_ENGINE", "INTERNAL_CPU_RSHIM", "NUM_OF_VFS"} { + stage = "nvconfig:" + card.address + ":" + parameter + text, err = remote.Run(ctx, stage, privilege+"mlxconfig -d "+card.address+" q "+parameter) + var value inventory.Observation + if err == nil { + value, err = parseNVConfig(text, parameter) + } + if err != nil { + value = unavailable(err, "mlxconfig") + } else { + c.EvidenceRefs = append(c.EvidenceRefs, add(stage, parameter)) + } + c.NVConfig[parameter] = value + } + if role == "dpus" { + stage = "privilege:" + card.address + text, err = remote.Run(ctx, stage, privilege+"mlxprivhost -d "+card.address+" q --full") + if err == nil { + c.Privilege, err = parsePrivilege(text) + } + if err != nil { + c.Privilege = unavailable(err, "mlxprivhost") + } else { + c.EvidenceRefs = append(c.EvidenceRefs, add(stage, "host-restrictions")) + } + } + } + h.Cards = append(h.Cards, c) + } +} + +func parseFirmware(text, bdf string) (inventory.Observation, inventory.Observation, error) { + fw, psid := missing("mlxfwmanager"), missing("mlxfwmanager") + seen := map[string]string{} + for _, line := range strings.Split(text, "\n") { + key, value, hasColon := strings.Cut(strings.TrimSpace(line), ":") + if hasColon && (key == "PCI Device Name" || key == "PSID") { + value = strings.TrimSpace(value) + if old, ok := seen[key]; ok && old != value { + return fw, psid, &Error{Code: "SSH_RUNTIME_INVALID"} + } + seen[key] = value + } + fields := strings.Fields(line) + if len(fields) >= 2 && fields[0] == "FW" { + if old, ok := seen["FW"]; ok && old != fields[1] { + return fw, psid, &Error{Code: "SSH_RUNTIME_INVALID"} + } + seen["FW"] = fields[1] + } + } + // An exact -d query is the binding. If a PCI address is reported, it must agree. + if address := seen["PCI Device Name"]; address != "" && !strings.EqualFold(address, bdf) && !strings.EqualFold("0000:"+address, bdf) { + return fw, psid, &Error{Code: "SSH_RUNTIME_INVALID"} + } + if v := seen["FW"]; versionToken.MatchString(v) { + fw = observed("mlxfwmanager", v) + } + if v := seen["PSID"]; versionToken.MatchString(v) { + psid = observed("mlxfwmanager", v) + } + if fw.State != "observed" && psid.State != "observed" { + return fw, psid, &Error{Code: "SSH_RUNTIME_INVALID"} + } + return fw, psid, nil +} + +func parseNVConfig(text, parameter string) (inventory.Observation, error) { + // Default mlxconfig q reports Next Boot, not necessarily the running state. + // Preserve this distinction rather than infer mode from a pending value. + next := strings.Contains(text, "Next Boot") + var value string + for _, line := range strings.Split(text, "\n") { + f := strings.Fields(line) + if len(f) > 0 && f[0] == parameter { + if len(f) != 2 || !versionToken.MatchString(f[1]) || value != "" { + return inventory.Observation{}, &Error{Code: "SSH_RUNTIME_INVALID"} + } + value = f[1] + } + } + if value == "" || !next { + return inventory.Observation{}, &Error{Code: "SSH_RUNTIME_INVALID"} + } + return observed("mlxconfig", "next-boot="+value), nil +} + +func parsePrivilege(text string) (inventory.Observation, error) { + host := "" + values := []string{} + seen := map[string]bool{} + levels := 0 + for _, line := range strings.Split(text, "\n") { + key, value, ok := strings.Cut(strings.TrimSpace(line), ":") + if !ok { + continue + } + key, value = strings.TrimSpace(key), strings.TrimSpace(value) + if key == "host index" { + if value == "" || strings.Trim(value, "0123456789") != "" { + return inventory.Observation{}, &Error{Code: "SSH_RUNTIME_INVALID"} + } + host = value + continue + } + if key != "level" && key != "disable_rshim" && key != "disable_tracer" && key != "disable_port_owner" && key != "disable_counter_rd" { + continue + } + id := "host[" + host + "]." + key + if host == "" || seen[id] { + return inventory.Observation{}, &Error{Code: "SSH_RUNTIME_INVALID"} + } + if key == "level" { + if value != "RESTRICTED" && value != "PRIVILEGED" { + return inventory.Observation{}, &Error{Code: "SSH_RUNTIME_INVALID"} + } + levels++ + } else if value != "TRUE" && value != "FALSE" { + return inventory.Observation{}, &Error{Code: "SSH_RUNTIME_INVALID"} + } + seen[id] = true + values = append(values, id+"="+value) + } + if levels == 0 { + return inventory.Observation{}, &Error{Code: "SSH_RUNTIME_INVALID"} + } + return observed("mlxprivhost", values...), nil +} diff --git a/pkg/dpf/sshprobe/hardware_test.go b/pkg/dpf/sshprobe/hardware_test.go new file mode 100644 index 00000000..bc9078ee --- /dev/null +++ b/pkg/dpf/sshprobe/hardware_test.go @@ -0,0 +1,80 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "context" + "fmt" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("hardware observations", func() { + It("queries each host card independently and retains specific missing-tool reasons", func() { + r := hostRunner() + r.values["firmware:0000:21:00.0"] = "PCI Device Name: 0000:21:00.0\nPSID: MT_000001\nVersions: Current Available\nFW 32.1.2 N/A\n" + r.errors["firmware:0000:41:00.0"] = &Error{Code: "TOOL_NOT_AVAILABLE", Stage: "firmware:0000:41:00.0", Tool: "mlxfwmanager"} + r.values["nvconfig:0000:21:00.0:NUM_OF_VFS"] = "Configurations: Next Boot\nNUM_OF_VFS 0\n" + d, _, err := collect(context.Background(), config.Target{Role: "hosts", Endpoint: "ssh://192.0.2.11:22"}, r) + Expect(err).NotTo(HaveOccurred()) + Expect(d.Runtime.Hardware.Cards).To(HaveLen(2)) + a, b := d.Runtime.Hardware.Cards[0], d.Runtime.Hardware.Cards[1] + Expect(a.Firmware.Values).To(Equal([]string{"32.1.2"})) + Expect(a.PSID.Values).To(Equal([]string{"MT_000001"})) + Expect(a.SerialNumber).To(Equal(testSerial)) + Expect(a.NVConfig["NUM_OF_VFS"].Values).To(Equal([]string{"next-boot=0"})) + Expect(a.Privilege.ReasonCode).To(Equal("NOT_APPLICABLE")) + Expect(b.Firmware.State).To(Equal("unavailable")) + Expect(b.Firmware.ReasonCode).To(Equal("TOOL_NOT_AVAILABLE")) + Expect(b.Firmware.Tool).To(Equal("mlxfwmanager")) + Expect(b.Firmware.Message).To(ContainSubstring("not available")) + Expect(strings.Join(r.calls, "\n")).NotTo(ContainSubstring("mlxprivhost")) + }) + It("collects ARM-local privileges without treating every ARM-visible NIC as the DPU", func() { + r := armRunner() + r.values["mft-version"] = "mlxfwmanager, mft 4.33.0-169, built on a date" + r.values["arm-pci"] = strings.ReplaceAll(testPCI, "0000:", "0002:") + r.values["privilege:0002:21:00.0"] = "host index : 0\nlevel : RESTRICTED\ndisable_rshim : TRUE\ndisable_tracer : TRUE\ndisable_port_owner : TRUE\ndisable_counter_rd : TRUE\n" + d, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22", SSH: &config.SSHAccess{Sudo: true}}, r) + Expect(err).NotTo(HaveOccurred()) + Expect(d.Runtime.Hardware.Cards).To(HaveLen(2)) + Expect(d.Runtime.Observations["mft-version"].Values).To(Equal([]string{"4.33.0-169"})) + Expect(d.Runtime.Observations["board-product"].Values).To(Equal([]string{"B4240"})) + Expect(d.Runtime.Hardware.Cards[0].Domain).To(Equal("arm-pci")) + Expect(d.Runtime.Hardware.Cards[0].SerialNumber).To(BeEmpty()) + Expect(d.Runtime.Hardware.Cards[0].Privilege.Values).To(ContainElement("host[0].level=RESTRICTED")) + Expect(d.Runtime.Hardware.Cards[1].Privilege.ReasonCode).To(Equal("NOT_APPLICABLE")) + Expect(r.calls).To(ContainElement("privilege:0002:21:00.0 sudo -n mlxprivhost -d 0002:21:00.0 q --full")) + }) + It("rejects firmware from a different PCI function and ambiguous tool output", func() { + for _, text := range []string{"PCI Device Name: 0000:22:00.0\nFW 32.1.2 N/A", "FW 32.1.2 N/A\nFW 32.1.3 N/A", "unrecognized text"} { + _, _, err := parseFirmware(text, "0000:21:00.0") + Expect(err).To(HaveOccurred()) + } + for _, text := range []string{"Configurations: Next Boot\nNUM_OF_VFS 2\nNUM_OF_VFS 3", "Configurations: Current Next Boot\nNUM_OF_VFS 2 3", "NUM_OF_VFS 2"} { + _, err := parseNVConfig(text, "NUM_OF_VFS") + Expect(err).To(HaveOccurred()) + } + _, err := parsePrivilege("level : RESTRICTED\n") + Expect(err).To(HaveOccurred()) + }) + It("retains required-tool failures in the partial identity result", func() { + r := armRunner() + r.errors["arm-fru"] = &Error{Code: "TOOL_NOT_AVAILABLE", Stage: "arm-fru", Tool: "ipmitool"} + d, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22"}, r) + Expect(err).To(HaveOccurred()) + Expect(d.Runtime.Observations["arm-fru"].Message).To(ContainSubstring("not available")) + Expect(d.Runtime.Observations["arm-fru"].Tool).To(Equal("ipmitool")) + }) + It("classifies diagnostics without persisting remote text", func() { + for text, code := range map[string]string{"sudo: mlxfwmanager: command not found": "TOOL_NOT_AVAILABLE", "sudo: a password is required": "PERMISSION_DENIED", "device operation not supported": "OPERATION_UNSUPPORTED", "opaque secret-value": "SSH_COMMAND_FAILED", "Status: Failed to open device": "DEVICE_ACCESS_FAILED"} { + err := commandFailure(fmt.Errorf("secret-value"), "firmware", "sudo -n mlxfwmanager --query", text) + Expect(err.Code).To(Equal(code)) + Expect(err.Tool).To(Equal("mlxfwmanager")) + Expect(Reason(err)).NotTo(ContainSubstring("secret-value")) + } + }) +}) diff --git a/pkg/dpf/sshprobe/network.go b/pkg/dpf/sshprobe/network.go new file mode 100644 index 00000000..606f7508 --- /dev/null +++ b/pkg/dpf/sshprobe/network.go @@ -0,0 +1,253 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "context" + "encoding/json" + "net" + "regexp" + "sort" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +const dnsmasqRead = `sh -c 'found=0; for f in /etc/dnsmasq.conf /etc/dnsmasq.d/*.conf; do if [ -e "$f" ]; then found=1; [ -r "$f" ] || exit 77; awk '\''/^[[:space:]]*(server=|no-resolv([[:space:]]|$))/ {print}'\'' "$f" || exit; fi; done; [ "$found" -eq 1 ] || exit 66'` + +func collectNetwork(ctx context.Context, remote runner, runtime *inventory.RuntimeObservation, add func(string, string) string) { + n := &inventory.NetworkObservation{Links: []inventory.NetworkLink{}, Neighbors: []inventory.Neighbor{}, LLDP: []inventory.LLDPNeighbor{}, Resolver: inventory.ResolverObservation{Owner: "unknown", Search: []string{}, PerLink: map[string][]string{}, Domains: map[string][]string{}, DNSMasqServers: []string{}}} + runtime.Network = n + for _, p := range []struct{ stage, command string }{ + {"network-links", "ip -j link show"}, {"network-link-details", "ip -j -d link show"}, {"neighbors", "ip -j neighbor show"}, {"resolver-path", "readlink -f /etc/resolv.conf"}, + {"resolver-dns", "resolvectl dns"}, {"resolver-domains", "resolvectl domain"}, {"dnsmasq", dnsmasqRead}, {"lldp", "lldpctl -f keyvalue"}, + } { + text, err := remote.Run(ctx, p.stage, p.command) + if err == nil { + switch p.stage { + case "network-links": + n.Links, err = parseLinks(text) + case "network-link-details": + var details []inventory.NetworkLink + details, err = parseLinks(text) + if err == nil { + err = mergeLinkDetails(n.Links, details) + } + case "neighbors": + n.Neighbors, err = parseNeighbors(text) + case "resolver-path": + value := strings.TrimSpace(text) + if !strings.HasPrefix(value, "/") || strings.ContainsAny(value, "\r\n\x00") { + err = &Error{Code: "SSH_RUNTIME_INVALID"} + break + } + n.Resolver.Path = value + switch { + case strings.Contains(value, "/systemd/resolve/"): + n.Resolver.Owner = "systemd-resolved" + case strings.Contains(value, "/NetworkManager/"): + n.Resolver.Owner = "NetworkManager" + case strings.Contains(value, "/resolvconf/"): + n.Resolver.Owner = "resolvconf" + default: + n.Resolver.Owner = "file" + } + case "resolver-dns": + n.Resolver.PerLink, err = parseResolved(text, true) + case "resolver-domains": + n.Resolver.Domains, err = parseResolved(text, false) + case "dnsmasq": + n.Resolver.DNSMasqServers, err = parseDNSMasq(text) + if err == nil { + noResolv := false + for _, line := range strings.Split(text, "\n") { + if strings.TrimSpace(line) == "no-resolv" { + noResolv = true + } + } + n.Resolver.DNSMasqNoResolv = &noResolv + } + case "lldp": + n.LLDP, err = parseLLDP(text) + } + } + if err != nil { + runtime.Observations[p.stage] = unavailable(err, commandTool(p.command)) + } else { + runtime.Observations[p.stage] = observed(commandTool(p.command)) + add(p.stage, p.stage) + } + } +} + +func parseLinks(text string) ([]inventory.NetworkLink, error) { + var rows []struct { + Name string `json:"ifname"` + MAC string `json:"address"` + Master string `json:"master"` + Info struct { + Kind string `json:"info_kind"` + } `json:"linkinfo"` + } + if err := json.Unmarshal([]byte(text), &rows); err != nil || rows == nil || len(rows) > 4096 { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + out := []inventory.NetworkLink{} + seen := map[string]bool{} + for _, r := range rows { + if r.Name == "" || seen[r.Name] { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + seen[r.Name] = true + // Non-Ethernet link-layer addresses (e.g. InfiniBand) are retained as facts. + out = append(out, inventory.NetworkLink{Name: r.Name, MAC: r.MAC, Master: r.Master, Kind: r.Info.Kind}) + } + return out, nil +} +func parseNeighbors(text string) ([]inventory.Neighbor, error) { + var rows []struct { + Address string `json:"dst"` + MAC string `json:"lladdr"` + Interface string `json:"dev"` + State []string `json:"state"` + } + if err := json.Unmarshal([]byte(text), &rows); err != nil || rows == nil || len(rows) > 8192 { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + out := []inventory.Neighbor{} + for _, r := range rows { + if net.ParseIP(r.Address) == nil || r.Interface == "" { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + if r.MAC != "" { + if mac, err := net.ParseMAC(r.MAC); err != nil || len(mac) != 6 { + continue + } + } + out = append(out, inventory.Neighbor{Address: r.Address, MAC: strings.ToLower(r.MAC), Interface: r.Interface, State: r.State}) + } + return out, nil +} + +var resolvedLine = regexp.MustCompile(`^(Global|Link [0-9]+ \(([^)]+)\)):\s*(.*)$`) + +func parseResolved(text string, addresses bool) (map[string][]string, error) { + out := map[string][]string{} + for _, line := range strings.Split(strings.TrimSpace(text), "\n") { + if strings.TrimSpace(line) == "" { + continue + } + m := resolvedLine.FindStringSubmatch(line) + if m == nil { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + key := m[2] + if key == "" { + key = "global" + } + if _, exists := out[key]; exists { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + values := strings.Fields(m[3]) + for _, v := range values { + if addresses { + ip, _, _ := strings.Cut(v, "%") + if net.ParseIP(ip) == nil { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + } + } + out[key] = values + } + if len(out) == 0 { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + return out, nil +} + +var dnsmasqValue = regexp.MustCompile(`^[A-Za-z0-9._:/#%~@+-]+$`) + +func parseDNSMasq(text string) ([]string, error) { + out := []string{} + for _, line := range strings.Split(text, "\n") { + line = strings.TrimSpace(line) + if line == "" || line == "no-resolv" || strings.HasPrefix(line, "no-resolv ") { + continue + } + value, ok := strings.CutPrefix(line, "server=") + if !ok || !dnsmasqValue.MatchString(value) { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + out = append(out, value) + } + return out, nil +} +func parseLLDP(text string) ([]inventory.LLDPNeighbor, error) { + byKey := map[string]inventory.LLDPNeighbor{} + for _, line := range strings.Split(text, "\n") { + key, value, ok := strings.Cut(strings.TrimSpace(line), "=") + if !ok { + continue + } + key, ok = strings.CutPrefix(key, "lldp.") + if !ok { + continue + } + for _, suffix := range []string{".chassis.name", ".port.ifname", ".port.descr"} { + iface, ok := strings.CutSuffix(key, suffix) + if !ok { + continue + } + if iface == "" || len(value) > 256 || strings.ContainsAny(value, "\x00\r\n") { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + n := byKey[iface] + n.Interface = iface + if suffix == ".chassis.name" { + if n.Chassis != "" && n.Chassis != value { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + n.Chassis = value + } else if suffix == ".port.ifname" || n.Port == "" { + n.Port = value + } + byKey[iface] = n + } + } + keys := []string{} + for k := range byKey { + keys = append(keys, k) + } + sort.Strings(keys) + out := []inventory.LLDPNeighbor{} + for _, k := range keys { + out = append(out, byKey[k]) + } + if len(out) == 0 && strings.TrimSpace(text) != "" { + return nil, &Error{Code: "SSH_RUNTIME_INVALID"} + } + return out, nil +} + +// Detailed iproute output is optional: some releases emit invalid JSON for +// VXLAN attributes. Base links remain useful without detailed link kinds. Verify +// a complete unchanged name/MAC/master projection before enriching those facts. +func mergeLinkDetails(base, details []inventory.NetworkLink) error { + if len(base) != len(details) { + return &Error{Code: "SSH_RUNTIME_INVALID", Stage: "network-link-details"} + } + byName := map[string]inventory.NetworkLink{} + for _, d := range details { + byName[d.Name] = d + } + for _, b := range base { + d, exists := byName[b.Name] + if !exists || d.MAC != b.MAC || d.Master != b.Master { + return &Error{Code: "SSH_RUNTIME_INVALID", Stage: "network-link-details"} + } + } + for i := range base { + base[i].Kind = byName[base[i].Name].Kind + } + return nil +} diff --git a/pkg/dpf/sshprobe/network_test.go b/pkg/dpf/sshprobe/network_test.go new file mode 100644 index 00000000..154c620c --- /dev/null +++ b/pkg/dpf/sshprobe/network_test.go @@ -0,0 +1,88 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "context" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("network observations", func() { + It("preserves actual bridges, upstream resolvers and candidate neighbors", func() { + r := armRunner() + r.values["network-links"] = `[{"ifname":"oob_net0","address":"02:00:00:00:00:41","master":"br0"},{"ifname":"br0","linkinfo":{"info_kind":"bridge"}}]` + r.values["resolver-path"] = "/run/systemd/resolve/stub-resolv.conf\n" + r.values["resolver-dns"] = "Global: 192.0.2.53\nLink 2 (oob_net0): 2001:db8::53\n" + r.values["resolver-domains"] = "Global:\nLink 2 (oob_net0): example.test ~.\n" + r.values["dns"] = "nameserver 127.0.0.53\nsearch example.test\n" + r.values["dnsmasq"] = "server=/example.test/192.0.2.53\nno-resolv\n" + r.values["neighbors"] = `[{"dst":"192.0.2.42","dev":"oob_net0","lladdr":"02:00:00:00:00:42","state":["STALE"]},{"dst":"192.0.2.43","dev":"oob_net0","state":["FAILED"]}]` + r.values["lldp"] = "lldp.oob_net0.chassis.name=switch-a\nlldp.oob_net0.port.ifname=swp1\n" + d, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22"}, r) + Expect(err).NotTo(HaveOccurred()) + n := d.Runtime.Network + Expect(n.Links[0].Master).To(Equal("br0")) + Expect(n.Links[1].Kind).To(Equal("bridge")) + Expect(n.Resolver.Owner).To(Equal("systemd-resolved")) + Expect(n.Resolver.PerLink["global"]).To(Equal([]string{"192.0.2.53"})) + Expect(n.Resolver.Search).To(Equal([]string{"example.test"})) + Expect(d.Runtime.DNS).To(Equal([]string{"127.0.0.53"})) + Expect(n.Resolver.DNSMasqServers).To(Equal([]string{"/example.test/192.0.2.53"})) + Expect(n.Neighbors).To(HaveLen(2)) + Expect(n.Neighbors[1].MAC).To(BeEmpty()) + Expect(n.LLDP[0].Chassis).To(Equal("switch-a")) + Expect(strings.Join(r.calls, "\n")).NotTo(ContainSubstring("192.0.2.43")) + }) + It("retains base link facts when detailed iproute VXLAN JSON is malformed", func() { + r := armRunner() + r.values["network-links"] = `[{"ifname":"oob_net0","address":"02:00:00:00:00:41","master":"br0"},{"ifname":"br0"}]` + r.values["network-link-details"] = `[{"ifname":"vxlan0","linkinfo":{"info_kind":"vxlan","info_data":{"fan_map":}}}]` + d, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22"}, r) + Expect(err).NotTo(HaveOccurred()) + Expect(d.Runtime.Network.Links).To(HaveLen(2)) + Expect(d.Runtime.Network.Links[0].MAC).To(Equal("02:00:00:00:00:41")) + Expect(d.Runtime.Network.Links[0].Master).To(Equal("br0")) + Expect(d.Runtime.Observations["network-links"].State).To(Equal("observed")) + Expect(d.Runtime.Observations["network-link-details"].ReasonCode).To(Equal("SSH_RUNTIME_INVALID")) + Expect(d.Runtime.Observations["network-link-details"].Message).NotTo(BeEmpty()) + }) + It("enriches base link kinds only when detailed observations still match", func() { + base, err := parseLinks(`[{"ifname":"br0","address":"02:00:00:00:00:01"}]`) + Expect(err).NotTo(HaveOccurred()) + details, err := parseLinks(`[{"ifname":"br0","address":"02:00:00:00:00:01","linkinfo":{"info_kind":"bridge"}}]`) + Expect(err).NotTo(HaveOccurred()) + Expect(mergeLinkDetails(base, details)).To(Succeed()) + Expect(base[0].Kind).To(Equal("bridge")) + details[0].MAC = "02:00:00:00:00:02" + Expect(mergeLinkDetails(base, details)).NotTo(Succeed()) + Expect(base[0].MAC).To(Equal("02:00:00:00:00:01")) + }) + It("does not interpret failed or malformed probes as empty successful evidence", func() { + r := armRunner() + r.values["neighbors"] = "null" + r.values["network-links"] = "bad-json" + r.errors["resolver-dns"] = &Error{Code: "TOOL_NOT_AVAILABLE", Stage: "resolver-dns", Tool: "resolvectl"} + d, _, err := collect(context.Background(), config.Target{Role: "dpus", Endpoint: "ssh://192.0.2.41:22"}, r) + Expect(err).NotTo(HaveOccurred()) + for _, stage := range []string{"neighbors", "network-links", "resolver-dns", "dnsmasq", "lldp"} { + Expect(d.Runtime.Observations[stage].State).To(Equal("unavailable")) + Expect(d.Runtime.Observations[stage].ReasonCode).NotTo(BeEmpty()) + Expect(d.Runtime.Observations[stage].Message).NotTo(BeEmpty()) + } + Expect(d.Runtime.Observations["resolver-dns"].Tool).To(Equal("resolvectl")) + }) + It("rejects invalid addresses and duplicate resolver records", func() { + for _, text := range []string{"Global: malformed", "Global: 192.0.2.53\nGlobal: 192.0.2.54"} { + _, err := parseResolved(text, true) + Expect(err).To(HaveOccurred()) + } + _, err := parseNeighbors(`[{"dst":"bad","dev":"eth0"}]`) + Expect(err).To(HaveOccurred()) + _, err = parseDNSMasq("server=credential with spaces") + Expect(err).To(HaveOccurred()) + }) +}) diff --git a/pkg/dpf/sshprobe/parse.go b/pkg/dpf/sshprobe/parse.go new file mode 100644 index 00000000..feb6a8d4 --- /dev/null +++ b/pkg/dpf/sshprobe/parse.go @@ -0,0 +1,240 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "encoding/json" + "net" + "regexp" + "sort" + "strconv" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +var ( + pciLine = regexp.MustCompile(`(?i)^([0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-7]) (.+) \[([0-9a-f]{4}):([0-9a-f]{4})\](?: .*)?$`) + uuidPattern = regexp.MustCompile(`(?i)^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`) +) + +func validSerial(s string) bool { + s = strings.TrimSpace(s) + if s == "" || len(s) > 256 || strings.ContainsAny(s, "\x00\r\n\t") { + return false + } + switch strings.ToLower(s) { + case "unknown", "n/a", "none", "default string", "to be filled by o.e.m.": + return false + } + return strings.Trim(s, "0- ") != "" +} + +func validUUID(s string) bool { + return uuidPattern.MatchString(s) && strings.Trim(strings.ToLower(s), "0-") != "" && strings.Trim(strings.ToLower(s), "f-") != "" +} + +// fields parses fixed-label utility output. Duplicate labels with different +// values are rejected so a later FRU or injected line cannot replace identity. +func fields(text string) (map[string]string, error) { + out := map[string]string{} + for _, line := range strings.Split(text, "\n") { + k, v, ok := strings.Cut(line, ":") + if !ok { + continue + } + k, v = strings.TrimSpace(k), strings.TrimSpace(v) + if k == "" { + continue + } + if before, exists := out[k]; exists && before != v { + return nil, &Error{Code: "SSH_IDENTITY_AMBIGUOUS", Stage: "duplicate-field"} + } + out[k] = v + } + return out, nil +} + +func armIdentity(dmi, fru string) (serial, model, manufacturer string, err error) { + f, err := fields(fru) + if err != nil { + return "", "", "", err + } + serial = strings.TrimSpace(dmi) + model, manufacturer = f["Product Name"], f["Product Manufacturer"] + if !validSerial(serial) || !validSerial(f["Product Serial"]) || serial != f["Product Serial"] { + return "", "", "", &Error{Code: "SSH_IDENTITY_CONFLICT", Stage: "arm-dmi-fru"} + } + if !strings.EqualFold(manufacturer, "nvidia") || !strings.Contains(strings.ToLower(model), "bluefield") { + return "", "", "", &Error{Code: "SSH_DPU_IDENTITY_UNSUPPORTED", Stage: "arm-fru"} + } + if board := f["Board Serial"]; board != "" && board != serial { + return "", "", "", &Error{Code: "SSH_IDENTITY_CONFLICT", Stage: "arm-board-fru"} + } + return serial, model, manufacturer, nil +} + +func bmcAddress(text string) (string, error) { + f, err := fields(text) + if err != nil { + return "", err + } + ip := net.ParseIP(f["IP Address"]) + mac, macErr := net.ParseMAC(f["MAC Address"]) + if ip == nil || ip.To4() == nil || ip.IsUnspecified() || ip.IsLoopback() || ip.IsMulticast() || macErr != nil || len(mac) != 6 { + return "", &Error{Code: "SSH_BMC_ADDRESS_UNAVAILABLE", Stage: "arm-ipmi-lan"} + } + return "https://" + ip.String(), nil +} + +type pciCard struct { + address, vendor, device, model string + functions []string +} + +func pciCards(text string) ([]pciCard, error) { + bySlot := map[string]pciCard{} + seen := map[string]bool{} + for _, line := range strings.Split(strings.TrimSpace(text), "\n") { + if strings.TrimSpace(line) == "" { + continue + } + m := pciLine.FindStringSubmatch(line) + if m == nil { + return nil, &Error{Code: "SSH_PCI_INVENTORY_INVALID", Stage: "host-pci"} + } + if !strings.EqualFold(m[3], "15b3") || (!strings.Contains(m[2], "[0200]") && !strings.Contains(m[2], "[0280]")) { + continue + } + bdf := strings.ToLower(m[1]) + if seen[bdf] { + return nil, &Error{Code: "SSH_PCI_INVENTORY_INVALID", Stage: "duplicate-pci-function"} + } + seen[bdf] = true + if len(seen) > 256 { + return nil, &Error{Code: "SSH_PCI_INVENTORY_LIMIT", Stage: "host-pci"} + } + slot := bdf[:len(bdf)-2] + old, ok := bySlot[slot] + functions := append(old.functions, bdf) + if !ok || bdf < old.address { + old = pciCard{address: bdf, vendor: strings.ToLower(m[3]), device: strings.ToLower(m[4]), model: m[2]} + } + old.functions = functions + bySlot[slot] = old + } + if len(bySlot) > 256 { + return nil, &Error{Code: "SSH_PCI_INVENTORY_LIMIT", Stage: "host-pci"} + } + out := make([]pciCard, 0, len(bySlot)) + for _, card := range bySlot { + sort.Strings(card.functions) + out = append(out, card) + } + sort.Slice(out, func(i, j int) bool { return out[i].address < out[j].address }) + return out, nil +} + +func vpd(text string) (serial, slot string, err error) { + for _, line := range strings.Split(text, "\n") { + line = strings.TrimSpace(line) + if strings.HasPrefix(line, "[SN] Serial number:") { + value := strings.TrimSpace(strings.TrimPrefix(line, "[SN] Serial number:")) + if serial != "" && serial != value { + return "", "", &Error{Code: "SSH_IDENTITY_AMBIGUOUS", Stage: "host-vpd"} + } + serial = value + } + if strings.HasPrefix(line, "Physical Slot:") { + slot = strings.TrimSpace(strings.TrimPrefix(line, "Physical Slot:")) + } + } + if !validSerial(serial) { + serial = "" + } + return serial, slot, nil +} + +func interfaces(text string) ([]inventory.Interface, error) { + var rows []struct { + Name string `json:"ifname"` + Index int `json:"ifindex"` + MTU int `json:"mtu"` + Addresses []struct { + Local string `json:"local"` + Prefix int `json:"prefixlen"` + } `json:"addr_info"` + } + if err := json.Unmarshal([]byte(text), &rows); err != nil { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "interfaces"} + } + if len(rows) > 4096 { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "interfaces"} + } + out := make([]inventory.Interface, 0, len(rows)) + for _, r := range rows { + if r.Name == "" || r.Index < 1 || r.MTU < 1 { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "interfaces"} + } + item := inventory.Interface{Name: r.Name, Index: r.Index, MTU: r.MTU, Addresses: []string{}} + for _, a := range r.Addresses { + address := a.Local + "/" + strconv.Itoa(a.Prefix) + if _, _, err := net.ParseCIDR(address); err != nil { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "interfaces"} + } + item.Addresses = append(item.Addresses, address) + } + out = append(out, item) + } + return out, nil +} + +func routes(text string, ifaces []inventory.Interface) ([]inventory.Route, error) { + var rows []struct { + Destination string `json:"dst"` + Gateway string `json:"gateway"` + Device string `json:"dev"` + Table json.RawMessage `json:"table"` + } + if err := json.Unmarshal([]byte(text), &rows); err != nil { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "routes"} + } + if len(rows) > 8192 { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "routes"} + } + indexes := map[string]int{} + for _, i := range ifaces { + indexes[i.Name] = i.Index + } + out := make([]inventory.Route, 0, len(rows)) + for _, r := range rows { + dst := r.Destination + if dst == "default" { + dst = "0.0.0.0/0" + } + if ip := net.ParseIP(dst); ip != nil { + if ip.To4() != nil { + dst += "/32" + } else { + dst += "/128" + } + } + if _, _, err := net.ParseCIDR(dst); err != nil { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "routes"} + } + if r.Gateway != "" && net.ParseIP(r.Gateway) == nil { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "routes"} + } + if indexes[r.Device] == 0 { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "routes"} + } + table := 254 + if len(r.Table) > 0 && string(r.Table) != "\"main\"" { + if err := json.Unmarshal(r.Table, &table); err != nil { + return nil, &Error{Code: "SSH_RUNTIME_INVALID", Stage: "routes"} + } + } + out = append(out, inventory.Route{Destination: dst, Gateway: r.Gateway, InterfaceIndex: indexes[r.Device], Table: table}) + } + return out, nil +} diff --git a/pkg/dpf/sshprobe/progress_test.go b/pkg/dpf/sshprobe/progress_test.go new file mode 100644 index 00000000..154f0d91 --- /dev/null +++ b/pkg/dpf/sshprobe/progress_test.go @@ -0,0 +1,116 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package sshprobe + +import ( + "bytes" + "context" + "encoding/json" + "sync" + "time" + + "github.com/go-logr/zapr" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "go.uber.org/zap" + "go.uber.org/zap/zapcore" + "go.uber.org/zap/zaptest/observer" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +type progressBuffer struct { + mu sync.Mutex + b bytes.Buffer +} + +func (b *progressBuffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.b.Write(p) +} +func (b *progressBuffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.b.String() +} + +type recheckBlocker struct { + remote runner + entered, release chan struct{} +} + +func (b recheckBlocker) Run(ctx context.Context, stage, command string) (string, error) { + if stage == "arm-dmi-recheck" { + close(b.entered) + select { + case <-b.release: + case <-ctx.Done(): + return "", &Error{Code: "SSH_TIMEOUT_OR_CANCELLED", Stage: stage} + } + } + return b.remote.Run(ctx, stage, command) +} + +var _ = Describe("secret-safe SSH stages", func() { + It("traces command metadata without command arguments, remote output, trust or credentials", func() { + server := startServer() + DeferCleanup(server.close) + for _, level := range []zapcore.Level{zapcore.DebugLevel, -2} { + core, records := observer.New(level) + ctx := log.IntoContext(context.Background(), zapr.NewLogger(zap.New(core))) + c, err := connect(ctx, "ssh://"+server.listener.Addr().String(), server.auth(), time.Second) + Expect(err).NotTo(HaveOccurred()) + _, err = c.Run(ctx, "host-vpd:private-id-canary", "query argument-canary") + Expect(err).NotTo(HaveOccurred()) + _, err = c.Run(ctx, "firmware", "fail") + Expect(Code(err)).To(Equal("SSH_COMMAND_FAILED")) + _ = c.Close() + if level == -2 { + stages := records.FilterMessage("SSH command stage completed").All() + Expect(stages).To(HaveLen(2)) + Expect(stages[0].ContextMap()).To(HaveKeyWithValue("stage", "host-vpd")) + Expect(stages[1].ContextMap()).To(HaveKeyWithValue("reasonCode", "SSH_COMMAND_FAILED")) + Expect(stages[1].ContextMap()).To(HaveKey("duration")) + } else { + Expect(records.Len()).To(BeZero()) + } + encoded, err := json.Marshal(records.All()) + Expect(err).NotTo(HaveOccurred()) + for _, secret := range []string{"test-password", "reader", "private-id-canary", "argument-canary", "remote sensitive diagnostic", "warning\\n", "observed\\n"} { + Expect(string(encoded)).NotTo(ContainSubstring(secret)) + } + } + }) + It("shows optional gaps and withholds identity success until a blocked recheck finishes", func() { + remote := recheckBlocker{remote: armRunner(), entered: make(chan struct{}), release: make(chan struct{})} + ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second) + DeferCleanup(cancel) + progress := &progressBuffer{} + ctx = ui.WithOutput(ctx, ui.NewWithWriter(progress)) + result := make(chan error, 1) + go func() { + _, _, err := collect(ctx, config.Target{Name: "DPU ARM", Role: "dpus", Endpoint: "ssh://192.0.2.41:22"}, remote) + result <- err + }() + Eventually(remote.entered).Should(BeClosed()) + Expect(progress.String()).To(ContainSubstring("[RUNNING] \"DPU ARM\"")) + Expect(progress.String()).To(ContainSubstring("[UNAVAILABLE]")) + Expect(progress.String()).To(ContainSubstring(": Physical identity recheck")) + Expect(progress.String()).NotTo(ContainSubstring(": Physical identity recheck verified")) + close(remote.release) + var err error + Eventually(result).Should(Receive(&err)) + Expect(err).NotTo(HaveOccurred()) + Expect(progress.String()).To(ContainSubstring(": Physical identity recheck verified")) + }) + It("counts missing nested card facts independently of successful tool reads", func() { + runtime := &inventory.RuntimeObservation{Observations: map[string]inventory.Observation{"tool": {State: "observed"}}, Hardware: &inventory.HardwareObservation{RShim: inventory.Observation{ReasonCode: "NOT_APPLICABLE"}, Cards: []inventory.CardHardware{{Firmware: inventory.Observation{State: "observed"}, PSID: inventory.Observation{State: "unknown"}, Privilege: inventory.Observation{ReasonCode: "NOT_APPLICABLE"}, NVConfig: map[string]inventory.Observation{"mode": {State: "unavailable"}}}}}} + observed, unavailable := optionalCounts(runtime) + Expect(observed).To(Equal(2)) + Expect(unavailable).To(Equal(2)) + }) +}) diff --git a/pkg/dpf/sshprobe/transport.go b/pkg/dpf/sshprobe/transport.go new file mode 100644 index 00000000..32466f01 --- /dev/null +++ b/pkg/dpf/sshprobe/transport.go @@ -0,0 +1,359 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package sshprobe observes explicitly selected Linux hosts and DPU ARM endpoints. +// It never installs software, changes configuration, or connects to an address +// learned from the remote system. +package sshprobe + +import ( + "bytes" + "context" + "errors" + "fmt" + "net" + "net/url" + "os" + "strings" + "sync" + "time" + + "golang.org/x/crypto/ssh" + "golang.org/x/crypto/ssh/knownhosts" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +const maxOutput = 1 << 20 + +// Error carries a stable diagnostic without remote output or credential material. +type Error struct { + Code string + Stage string + Tool string +} + +func (e *Error) Error() string { return e.Code + ": " + e.Stage } + +// Code classifies errors without exposing SSH library or remote error text. +func Code(err error) string { + var e *Error + if errors.As(err, &e) { + return e.Code + } + return "SSH_OBSERVATION_FAILED" +} + +type credentials struct { + user, password string + key, knownHosts []byte +} + +type client struct { + ssh *ssh.Client + stop func() bool + timeout time.Duration +} + +// reconnectingRunner owns a single SSH session and reconnects only after a +// timed-out command has closed that session. It never retries the command that +// timed out; the next requested read starts on a newly authenticated session. +type reconnectingRunner struct { + endpoint string + auth credentials + timeout time.Duration + client *client + reconnect bool + reconnectErr error + closed bool +} + +func newReconnectingRunner(ctx context.Context, endpoint string, auth credentials, timeout time.Duration) (*reconnectingRunner, error) { + auth.key = bytes.Clone(auth.key) + auth.knownHosts = bytes.Clone(auth.knownHosts) + c, err := connect(ctx, endpoint, auth, timeout) + if err != nil { + clear(auth.key) + clear(auth.knownHosts) + auth.password = "" + return nil, err + } + return &reconnectingRunner{endpoint: endpoint, auth: auth, timeout: timeout, client: c}, nil +} + +func (r *reconnectingRunner) Run(ctx context.Context, stage, command string) (string, error) { + if r.closed { + return "", &Error{Code: "SSH_COMMAND_FAILED", Stage: stage, Tool: commandTool(command)} + } + if err := ctx.Err(); err != nil { + return "", &Error{Code: "SSH_TIMEOUT_OR_CANCELLED", Stage: stage, Tool: commandTool(command)} + } + if r.reconnectErr != nil { + return "", r.reconnectErr + } + if r.reconnect { + if r.client != nil { + _ = r.client.Close() + } + c, err := connect(ctx, r.endpoint, r.auth, r.timeout) + if err != nil { + r.reconnectErr = err + return "", err + } + r.client = c + r.reconnect = false + } + text, err := r.client.Run(ctx, stage, command) + if Code(err) == "SSH_TIMEOUT_OR_CANCELLED" && ctx.Err() == nil { + // Run's private command deadline expired. Its cancellation hook closed + // this transport to interrupt an unresponsive channel; reconnect before + // the next independent read, with the original endpoint and trust. + r.reconnect = true + } + return text, err +} + +func (r *reconnectingRunner) Close() error { + if r.closed { + return nil + } + r.closed = true + clear(r.auth.key) + clear(r.auth.knownHosts) + r.auth.password = "" + if r.client != nil { + return r.client.Close() + } + return nil +} + +// hostKeyCallback loads a stable snapshot through the upstream known_hosts +// parser, preserving its hashed-host, wildcard, CA and revocation semantics. +// The temporary file contains public host keys only and is removed before dial. +func hostKeyCallback(data []byte) (ssh.HostKeyCallback, error) { + if len(bytes.TrimSpace(data)) == 0 || len(data) > maxOutput { + return nil, &Error{Code: "SSH_TRUST_INVALID", Stage: "known-hosts"} + } + f, err := os.CreateTemp("", "l8k-known-hosts-*") + if err != nil { + return nil, &Error{Code: "SSH_TRUST_INVALID", Stage: "known-hosts"} + } + defer os.Remove(f.Name()) + if _, err = f.Write(data); err != nil { + _ = f.Close() + return nil, &Error{Code: "SSH_TRUST_INVALID", Stage: "known-hosts"} + } + if err = f.Close(); err != nil { + return nil, &Error{Code: "SSH_TRUST_INVALID", Stage: "known-hosts"} + } + callback, err := knownhosts.New(f.Name()) + if err != nil { + return nil, &Error{Code: "SSH_TRUST_INVALID", Stage: "known-hosts"} + } + return callback, nil +} + +// preferredHostKeyAlgorithms avoids negotiating an unpinned server key when a +// different advertised key is already trusted. Host matching and revocation +// decisions belong to the upstream callback, including hashed host entries. +// Certificates remain available and are verified by that same callback: the +// signing CA's algorithm does not determine the host certificate's key type. +func preferredHostKeyAlgorithms(ctx context.Context, data []byte, verify ssh.HostKeyCallback, address string, remote net.Addr) ([]string, error) { + preferred := map[string]bool{} + seen := map[string]bool{} + for _, line := range bytes.Split(data, []byte{'\n'}) { + if ctx.Err() != nil { + return nil, &Error{Code: "SSH_TIMEOUT_OR_CANCELLED", Stage: "known-host-algorithms"} + } + fields := bytes.Fields(line) + if len(fields) < 3 || fields[0][0] == '#' || fields[0][0] == '@' { + continue + } + // New already validated the file. Parse just its key fields so arbitrary + // multi-word comments do not change ParseKnownHosts field-count handling. + _, _, key, _, _, err := ssh.ParseKnownHosts(bytes.Join(fields[:3], []byte{' '})) + if err != nil { + return nil, &Error{Code: "SSH_TRUST_INVALID", Stage: "known-host-algorithms"} + } + blob := string(key.Marshal()) + if seen[blob] { + continue + } + seen[blob] = true + if verify(address, remote, key) != nil { + continue + } + switch key.Type() { + case ssh.KeyAlgoRSA: + preferred[ssh.KeyAlgoRSASHA256], preferred[ssh.KeyAlgoRSASHA512] = true, true + default: + preferred[key.Type()] = true + } + } + supported := ssh.SupportedAlgorithms().HostKeys + result := make([]string, 0, len(supported)) + for _, a := range supported { + if preferred[a] { + result = append(result, a) + } + } + for _, a := range supported { + if !preferred[a] { + result = append(result, a) + } + } + return result, nil +} + +func connect(ctx context.Context, endpoint string, auth credentials, timeout time.Duration) (connected *client, err error) { + started := time.Now() + logger := log.FromContext(ctx) + logger.V(2).Info("SSH connection negotiation started") + defer func() { + reason := "" + if err != nil { + reason = Code(err) + } + logger.V(2).Info("SSH connection negotiation completed", "reasonCode", reason, "duration", time.Since(started)) + }() + u, err := url.Parse(endpoint) + if err != nil || u.Scheme != "ssh" || u.User != nil || u.Path != "" || u.RawQuery != "" || u.Fragment != "" || u.Hostname() == "" || u.Port() == "" || timeout <= 0 { + return nil, &Error{Code: "SSH_CONFIGURATION_INVALID", Stage: "endpoint"} + } + verify, err := hostKeyCallback(auth.knownHosts) + if err != nil { + return nil, err + } + if auth.user == "" || (auth.password == "") == (len(auth.key) == 0) { + return nil, &Error{Code: "SSH_CONFIGURATION_INVALID", Stage: "credential"} + } + var method ssh.AuthMethod + if len(auth.key) > 0 { + signer, parseErr := ssh.ParsePrivateKey(auth.key) + if parseErr != nil { + return nil, &Error{Code: "SSH_CONFIGURATION_INVALID", Stage: "private-key"} + } + method = ssh.PublicKeys(signer) + } else { + method = ssh.Password(auth.password) + } + dialCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + conn, err := (&net.Dialer{}).DialContext(dialCtx, "tcp", u.Host) + if err != nil { + return nil, &Error{Code: "SSH_CONNECT_FAILED", Stage: "dial"} + } + stop := context.AfterFunc(ctx, func() { _ = conn.Close() }) + failed := true + defer func() { + if failed { + stop() + _ = conn.Close() + } + }() + deadline, _ := dialCtx.Deadline() + if err = conn.SetDeadline(deadline); err != nil { + return nil, &Error{Code: "SSH_CONNECT_FAILED", Stage: "deadline"} + } + algorithms, err := preferredHostKeyAlgorithms(dialCtx, auth.knownHosts, verify, u.Host, conn.RemoteAddr()) + if err != nil { + return nil, err + } + trustFailed := false + cfg := &ssh.ClientConfig{User: auth.user, Auth: []ssh.AuthMethod{method}, HostKeyAlgorithms: algorithms, HostKeyCallback: func(host string, remote net.Addr, key ssh.PublicKey) error { + err := verify(host, remote, key) + trustFailed = err != nil + return err + }} + cc, channels, requests, err := ssh.NewClientConn(conn, u.Host, cfg) + if err != nil { + code := "SSH_AUTHENTICATION_OR_HANDSHAKE_FAILED" + if trustFailed { + code = "SSH_HOST_KEY_REJECTED" + } + return nil, &Error{Code: code, Stage: "handshake"} + } + if err = conn.SetDeadline(time.Time{}); err != nil { + _ = cc.Close() + return nil, &Error{Code: "SSH_CONNECT_FAILED", Stage: "deadline"} + } + failed = false + return &client{ssh: ssh.NewClient(cc, channels, requests), stop: stop, timeout: timeout}, nil +} + +func (c *client) Close() error { + c.stop() + return c.ssh.Close() +} + +type commandOutput struct { + mu sync.Mutex + stdout, stderr bytes.Buffer + size int + exceeded bool +} +type outputWriter struct { + output *commandOutput + stderr bool +} + +func (w outputWriter) Write(p []byte) (int, error) { + o := w.output + o.mu.Lock() + defer o.mu.Unlock() + if len(p) > maxOutput-o.size { + o.exceeded = true + return 0, fmt.Errorf("SSH output limit exceeded") + } + o.size += len(p) + if w.stderr { + return o.stderr.Write(p) + } + return o.stdout.Write(p) +} + +func (c *client) Run(ctx context.Context, stage, command string) (text string, err error) { + started := time.Now() + // Stage suffixes contain per-device identifiers; commands and remote output + // are intentionally excluded, including on failure. + name, _, _ := strings.Cut(stage, ":") + logger := log.FromContext(ctx) + logger.V(2).Info("SSH command stage started", "stage", name, "tool", commandTool(command)) + defer func() { + reason := "" + if err != nil { + reason = Code(err) + } + logger.V(2).Info("SSH command stage completed", "stage", name, "tool", commandTool(command), "reasonCode", reason, "duration", time.Since(started)) + }() + ctx, cancel := context.WithTimeout(ctx, c.timeout) + defer cancel() + // Closing the connection also interrupts NewSession/Start if the server stops + // responding. A timed-out connection is not reused for later observations. + stop := context.AfterFunc(ctx, func() { _ = c.ssh.Close() }) + defer stop() + session, err := c.ssh.NewSession() + if err != nil { + code := "SSH_COMMAND_FAILED" + if ctx.Err() != nil { + code = "SSH_TIMEOUT_OR_CANCELLED" + } + return "", &Error{Code: code, Stage: stage, Tool: commandTool(command)} + } + defer session.Close() + output := &commandOutput{} + session.Stdout = outputWriter{output: output} + session.Stderr = outputWriter{output: output, stderr: true} + err = session.Run("LC_ALL=C " + command) + output.mu.Lock() + defer output.mu.Unlock() + if ctx.Err() != nil { + return "", &Error{Code: "SSH_TIMEOUT_OR_CANCELLED", Stage: stage, Tool: commandTool(command)} + } + if output.exceeded { + return "", &Error{Code: "SSH_OUTPUT_LIMIT", Stage: stage, Tool: commandTool(command)} + } + if err != nil { + return "", commandFailure(err, stage, command, output.stdout.String()+"\n"+output.stderr.String()) + } + return output.stdout.String(), nil +} diff --git a/pkg/dpf/sshprobe/transport_test.go b/pkg/dpf/sshprobe/transport_test.go new file mode 100644 index 00000000..40e3fc87 --- /dev/null +++ b/pkg/dpf/sshprobe/transport_test.go @@ -0,0 +1,350 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package sshprobe + +import ( + "bytes" + "context" + "crypto/ecdsa" + "crypto/ed25519" + "crypto/elliptic" + "crypto/rand" + "crypto/rsa" + "encoding/pem" + "fmt" + "net" + "strings" + "sync" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "golang.org/x/crypto/ssh" + "golang.org/x/crypto/ssh/knownhosts" +) + +type testServer struct { + listener net.Listener + key ssh.Signer + private ed25519.PrivateKey + extraKeys []ssh.Signer + alternateKey ssh.Signer + rekeyAfter int + rejectAfter int + onCommand func(ssh.Channel, string) bool + mu sync.Mutex + connections []net.Conn + wg sync.WaitGroup + closing bool +} + +func startServer(extraKeys ...ssh.Signer) *testServer { + _, private, err := ed25519.GenerateKey(rand.Reader) + Expect(err).NotTo(HaveOccurred()) + key, err := ssh.NewSignerFromKey(private) + Expect(err).NotTo(HaveOccurred()) + listener, err := net.Listen("tcp", "127.0.0.1:0") + Expect(err).NotTo(HaveOccurred()) + s := &testServer{listener: listener, key: key, private: private, extraKeys: extraKeys, connections: []net.Conn{}} + s.wg.Add(1) + go func() { + defer s.wg.Done() + for { + c, err := listener.Accept() + if err != nil { + return + } + s.mu.Lock() + if s.closing { + s.mu.Unlock() + _ = c.Close() + return + } + s.connections = append(s.connections, c) + connection := len(s.connections) + s.wg.Add(1) + s.mu.Unlock() + go func() { defer s.wg.Done(); defer c.Close(); s.serve(c, connection) }() + } + }() + return s +} + +func (s *testServer) serve(conn net.Conn, connection int) { + s.mu.Lock() + serverKey := s.key + if s.rekeyAfter > 0 && connection >= s.rekeyAfter { + serverKey = s.alternateKey + } + rejectAuth := s.rejectAfter > 0 && connection >= s.rejectAfter + onCommand := s.onCommand + s.mu.Unlock() + cfg := &ssh.ServerConfig{PublicKeyCallback: func(meta ssh.ConnMetadata, key ssh.PublicKey) (*ssh.Permissions, error) { + if rejectAuth { + return nil, fmt.Errorf("denied") + } + if meta.User() == "reader" && string(key.Marshal()) == string(s.key.PublicKey().Marshal()) { + return nil, nil + } + return nil, fmt.Errorf("denied") + }, PasswordCallback: func(meta ssh.ConnMetadata, password []byte) (*ssh.Permissions, error) { + if rejectAuth { + return nil, fmt.Errorf("denied") + } + if meta.User() == "reader" && string(password) == "test-password" { + return nil, nil + } + return nil, fmt.Errorf("denied") + }} + cfg.AddHostKey(serverKey) + for _, key := range s.extraKeys { + cfg.AddHostKey(key) + } + server, chans, requests, err := ssh.NewServerConn(conn, cfg) + if err != nil { + return + } + defer server.Close() + go ssh.DiscardRequests(requests) + var sessions sync.WaitGroup + defer sessions.Wait() + for channel := range chans { + if channel.ChannelType() != "session" { + _ = channel.Reject(ssh.UnknownChannelType, "unsupported") + continue + } + ch, reqs, err := channel.Accept() + if err != nil { + return + } + sessions.Add(1) + go func() { + defer sessions.Done() + defer ch.Close() + for req := range reqs { + if req.Type != "exec" { + _ = req.Reply(false, nil) + continue + } + var payload struct{ Command string } + if ssh.Unmarshal(req.Payload, &payload) != nil { + _ = req.Reply(false, nil) + return + } + _ = req.Reply(true, nil) + if onCommand != nil && onCommand(ch, payload.Command) { + return + } + switch payload.Command { + case "LC_ALL=C slow": + for range reqs { + } + return + case "LC_ALL=C flood": + _, _ = ch.Write([]byte(strings.Repeat("x", maxOutput+1))) + case "LC_ALL=C missing": + _, _ = ch.SendRequest("exit-status", false, ssh.Marshal(struct{ Status uint32 }{127})) + return + case "LC_ALL=C not-executable": + _, _ = ch.SendRequest("exit-status", false, ssh.Marshal(struct{ Status uint32 }{126})) + return + case "LC_ALL=C sudo -n mlxfwmanager --query": + _, _ = ch.Stderr().Write([]byte("sh: 1: sudo: not found")) + _, _ = ch.SendRequest("exit-status", false, ssh.Marshal(struct{ Status uint32 }{127})) + return + case "LC_ALL=C fail": + _, _ = ch.Stderr().Write([]byte("remote sensitive diagnostic")) + _, _ = ch.SendRequest("exit-status", false, ssh.Marshal(struct{ Status uint32 }{7})) + return + default: + _, _ = ch.Write([]byte("observed\n")) + _, _ = ch.Stderr().Write([]byte("warning\n")) + } + _, _ = ch.SendRequest("exit-status", false, ssh.Marshal(struct{ Status uint32 }{0})) + return + } + }() + } +} + +func (s *testServer) close() { + s.mu.Lock() + s.closing = true + _ = s.listener.Close() + for _, c := range s.connections { + _ = c.Close() + } + s.mu.Unlock() + s.wg.Wait() +} +func (s *testServer) auth() credentials { + return credentials{user: "reader", password: "test-password", knownHosts: []byte(knownhosts.Line([]string{s.listener.Addr().String()}, s.key.PublicKey()) + "\n")} +} + +var _ = Describe("bounded SSH transport", func() { + var server *testServer + BeforeEach(func() { server = startServer() }) + AfterEach(func() { server.close() }) + It("verifies host identity before authentication and keeps stdout separate", func() { + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), server.auth(), time.Second) + Expect(err).NotTo(HaveOccurred()) + defer c.Close() + out, err := c.Run(context.Background(), "query", "query") + Expect(err).NotTo(HaveOccurred()) + Expect(out).To(Equal("observed\n")) + }) + It("accepts hashed known-hosts entries through the upstream parser", func() { + auth := server.auth() + auth.knownHosts = []byte(knownhosts.HashHostname(knownhosts.Normalize(server.listener.Addr().String())) + " " + strings.TrimSpace(string(ssh.MarshalAuthorizedKey(server.key.PublicKey()))) + "\n") + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(err).NotTo(HaveOccurred()) + c.Close() + }) + It("selects the pinned Ed25519 key on a server also offering ECDSA and RSA", func() { + ecdsaKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + Expect(err).NotTo(HaveOccurred()) + ecdsaSigner, err := ssh.NewSignerFromKey(ecdsaKey) + Expect(err).NotTo(HaveOccurred()) + rsaKey, err := rsa.GenerateKey(rand.Reader, 2048) + Expect(err).NotTo(HaveOccurred()) + rsaSigner, err := ssh.NewSignerFromKey(rsaKey) + Expect(err).NotTo(HaveOccurred()) + server.close() + server = startServer(ecdsaSigner, rsaSigner) + for _, hashed := range []bool{false, true} { + auth := server.auth() + if hashed { + auth.knownHosts = []byte(knownhosts.HashHostname(knownhosts.Normalize(server.listener.Addr().String())) + " " + strings.TrimSpace(string(ssh.MarshalAuthorizedKey(server.key.PublicKey()))) + "\n") + } + auth.knownHosts = append([]byte(knownhosts.Line([]string{"other.example"}, ecdsaSigner.PublicKey())+"\n"), bytes.TrimSpace(auth.knownHosts)...) + auth.knownHosts = append(auth.knownHosts, []byte(" pinned host key with a multi word comment\n")...) + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(err).NotTo(HaveOccurred()) + c.Close() + } + auth := server.auth() + auth.knownHosts = []byte(knownhosts.Line([]string{server.listener.Addr().String()}, rsaSigner.PublicKey()) + "\n") + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(err).NotTo(HaveOccurred()) + c.Close() + }) + It("preserves host certificate authority verification", func() { + _, caPrivate, err := ed25519.GenerateKey(rand.Reader) + Expect(err).NotTo(HaveOccurred()) + ca, err := ssh.NewSignerFromKey(caPrivate) + Expect(err).NotTo(HaveOccurred()) + certificate := &ssh.Certificate{Key: server.key.PublicKey(), CertType: ssh.HostCert, ValidPrincipals: []string{"127.0.0.1"}, ValidAfter: uint64(time.Now().Add(-time.Minute).Unix()), ValidBefore: uint64(time.Now().Add(time.Minute).Unix())} + Expect(certificate.SignCert(rand.Reader, ca)).To(Succeed()) + signer, err := ssh.NewCertSigner(certificate, server.key) + Expect(err).NotTo(HaveOccurred()) + server.close() + server = startServer(signer) + auth := server.auth() + auth.knownHosts = []byte("@cert-authority " + knownhosts.Line([]string{server.listener.Addr().String()}, ca.PublicKey()) + "\n") + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(err).NotTo(HaveOccurred()) + c.Close() + }) + It("does not promote revoked pinned keys", func() { + auth := server.auth() + auth.knownHosts = append(auth.knownHosts, []byte("@revoked "+knownhosts.Line([]string{server.listener.Addr().String()}, server.key.PublicKey())+"\n")...) + _, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(Code(err)).To(Equal("SSH_HOST_KEY_REJECTED")) + }) + It("uses only the explicitly provided private key", func() { + block, err := ssh.MarshalPrivateKey(server.private, "test") + Expect(err).NotTo(HaveOccurred()) + auth := server.auth() + auth.password = "" + auth.key = pem.EncodeToMemory(block) + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(err).NotTo(HaveOccurred()) + c.Close() + auth.key = []byte("invalid-private-key") + _, err = connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(Code(err)).To(Equal("SSH_CONFIGURATION_INVALID")) + Expect(err.Error()).NotTo(ContainSubstring("invalid-private-key")) + }) + It("rejects unknown and changed host keys", func() { + _, private, err := ed25519.GenerateKey(rand.Reader) + Expect(err).NotTo(HaveOccurred()) + other, err := ssh.NewSignerFromKey(private) + Expect(err).NotTo(HaveOccurred()) + for _, entry := range []string{knownhosts.Line([]string{"other.example"}, server.key.PublicKey()), knownhosts.Line([]string{server.listener.Addr().String()}, other.PublicKey())} { + auth := server.auth() + auth.knownHosts = []byte(entry + "\n") + _, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(Code(err)).To(Equal("SSH_HOST_KEY_REJECTED")) + } + }) + It("does not expose rejected credentials or remote stderr", func() { + auth := server.auth() + auth.password = "do-not-print" + _, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), auth, time.Second) + Expect(err).To(HaveOccurred()) + Expect(err.Error()).NotTo(ContainSubstring(auth.password)) + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), server.auth(), time.Second) + Expect(err).NotTo(HaveOccurred()) + defer c.Close() + _, err = c.Run(context.Background(), "failure", "fail") + Expect(Code(err)).To(Equal("SSH_COMMAND_FAILED")) + Expect(err.Error()).NotTo(ContainSubstring("sensitive")) + }) + It("distinguishes missing executables and missing sudo without remote output", func() { + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), server.auth(), time.Second) + Expect(err).NotTo(HaveOccurred()) + defer c.Close() + for command, code := range map[string]string{"missing": "TOOL_NOT_AVAILABLE", "not-executable": "TOOL_NOT_EXECUTABLE", "sudo -n mlxfwmanager --query": "TOOL_NOT_AVAILABLE"} { + output, err := c.Run(context.Background(), "query", command) + Expect(output).To(BeEmpty()) + Expect(Code(err)).To(Equal(code)) + if strings.HasPrefix(command, "sudo") { + Expect(err.(*Error).Tool).To(Equal("sudo")) + } + } + }) + It("bounds command output", func() { + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), server.auth(), time.Second) + Expect(err).NotTo(HaveOccurred()) + defer c.Close() + _, err = c.Run(context.Background(), "large", "flood") + Expect(Code(err)).To(Equal("SSH_OUTPUT_LIMIT")) + }) + It("cancels a command and closes its transport", func() { + c, err := connect(context.Background(), "ssh://"+server.listener.Addr().String(), server.auth(), time.Second) + Expect(err).NotTo(HaveOccurred()) + defer c.Close() + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + defer cancel() + start := time.Now() + _, err = c.Run(ctx, "slow", "slow") + Expect(Code(err)).To(Equal("SSH_TIMEOUT_OR_CANCELLED")) + Expect(time.Since(start)).To(BeNumerically("<", time.Second)) + }) + It("bounds an unresponsive handshake", func() { + listener, err := net.Listen("tcp", "127.0.0.1:0") + Expect(err).NotTo(HaveOccurred()) + defer listener.Close() + done := make(chan struct{}) + go func() { + conn, e := listener.Accept() + if e == nil { + defer conn.Close() + buf := make([]byte, 1024) + for { + if _, e = conn.Read(buf); e != nil { + break + } + } + } + close(done) + }() + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + defer cancel() + start := time.Now() + _, err = connect(ctx, "ssh://"+listener.Addr().String(), server.auth(), time.Second) + Expect(err).To(HaveOccurred()) + Expect(time.Since(start)).To(BeNumerically("<", time.Second)) + Eventually(done).Should(BeClosed()) + }) +}) diff --git a/pkg/networkoperatorplugin/target_scope_test.go b/pkg/networkoperatorplugin/target_scope_test.go index 97ee509a..1b263e52 100644 --- a/pkg/networkoperatorplugin/target_scope_test.go +++ b/pkg/networkoperatorplugin/target_scope_test.go @@ -322,8 +322,9 @@ var _ = ginkgo.Describe("hardware target scope", func() { }) ginkgo.DescribeTable("bounds temporary scope retries by caller context", func(cancelDuringRead bool) { desired := scopePolicy(map[string]string{"cohort": "gpu"}, "source") - pollCtx, cancel := context.WithTimeout(ctx, 30*time.Millisecond) - defer cancel() + // Explicit cancellation must not race a deadline under a loaded test runner. + var pollCtx context.Context + var cancel context.CancelFunc reads, validations := 0, 0 c := fake.NewClientBuilder().WithInterceptorFuncs(interceptor.Funcs{Get: func(context.Context, client.WithWatch, client.ObjectKey, client.Object, ...client.GetOption) error { reads++ @@ -337,13 +338,20 @@ var _ = ginkgo.Describe("hardware target scope", func() { validations++ return crstate.Result{State: crstate.StateSuccess}, nil }) + if cancelDuringRead { + pollCtx, cancel = context.WithCancel(ctx) + } else { + pollCtx, cancel = context.WithTimeout(ctx, 30*time.Millisecond) + } + defer cancel() err := pollUntilTerminalWithRetryableErrorTimeout(pollCtx, c, registry, desired, "policy", "", 0, time.Millisecond, cfg) if cancelDuringRead { gomega.Expect(err).To(gomega.MatchError(context.Canceled)) gomega.Expect(reads).To(gomega.Equal(1)) } else { gomega.Expect(err).To(gomega.MatchError(context.DeadlineExceeded)) - gomega.Expect(reads).To(gomega.BeNumerically(">", 1)) + // Retry counts depend on scheduling before the deadline; the preceding + // test proves retry behavior without relying on a wall-clock window. } gomega.Expect(validations).To(gomega.BeZero()) }, diff --git a/pkg/target/dpf/driver.go b/pkg/target/dpf/driver.go new file mode 100644 index 00000000..ea8cdb74 --- /dev/null +++ b/pkg/target/dpf/driver.go @@ -0,0 +1,156 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package dpf binds the independent DPU-plane discovery CLI to typed services. +package dpf + +import ( + "context" + "encoding/json" + "fmt" + "io" + "os" + "os/signal" + "path/filepath" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/artifacts" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/catalog" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/probe" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/redfish" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/sshprobe" + apperrors "github.com/nvidia/k8s-launch-kit/pkg/errors" + "github.com/nvidia/k8s-launch-kit/pkg/target" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +// Descriptor exposes the inventory preview, keeping provisioning unavailable. +func Descriptor() target.Descriptor { + phases := map[target.Phase]target.Capability{} + for _, p := range append(target.PublicPhases(), target.Pipeline) { + phases[p] = target.Capability{Available: false, Reason: "DPF provisioning is not implemented in this build"} + } + phases[target.Discover] = target.Capability{Available: true, Reason: "experimental inventory discovery; management probes require qualified release assets; hardware catalog qualification pending"} + return target.Descriptor{Name: target.DPF, Description: "Independent DPF inventory discovery preview", Phases: phases} +} + +type Request struct { + UserConfig string + OutputPath string + ExplicitOutput bool + Stdout io.Writer + Stderr io.Writer +} +type driver struct{ request Request } + +func NewDiscoverAdapter(r Request) target.Driver { return driver{request: r} } +func (d driver) Descriptor() target.Descriptor { return Descriptor() } +func (d driver) Bind(inv target.Invocation) (target.Operation, error) { + if inv.Phase != target.Discover { + return nil, &target.PhaseUnavailableError{Name: target.DPF, Phase: inv.Phase, Reason: Descriptor().Capability(inv.Phase).Reason} + } + if inv.Execution.DryRun { + return nil, fmt.Errorf("DPF discovery uses discovery.probes.mode: inventory to omit probes") + } + r := d.request + if r.Stdout == nil { + r.Stdout = os.Stdout + } + if r.Stderr == nil { + r.Stderr = os.Stderr + } + return target.OperationFunc(func(parent context.Context) error { + ctx, stop := signal.NotifyContext(parent, terminationSignals()...) + defer stop() + progress := ui.NewSilent() + if !inv.Output.Quiet { + progress = ui.NewWithWriter(r.Stderr) + } + ctx = ui.WithOutput(ctx, progress) + logger := log.FromContext(ctx).WithName("dpf-discovery") + ctx = log.IntoContext(ctx, logger) + report := discovery.NewReport() + var store *artifacts.Store + fail := func(code int, message string) { + report.Execution = "failed" + report.Errors = append(report.Errors, inventory.Problem{Code: "INVALID_CONFIGURATION", Message: message, ExitCode: code}) + progress.Info("[FAIL] Discovery preparation: %s", displayText(message)) + } + progress.Info("[RUNNING] Configuration: validating site inputs and access settings") + input, err := config.Load(r.UserConfig) + if err != nil { + fail(2, err.Error()) + } else { + resolved, err := config.Resolve(input) + if err != nil { + fail(2, err.Error()) + } else if err = validateManagement(resolved); err != nil { + fail(2, err.Error()) + } else { + // Unqualified image assets reject management mode before any remote operation. + assets := catalog.ProbeAssets() + if resolved.Mode == "management" && assets.Validate() != nil { + fail(2, "management probes unavailable until the pinned diagnostic image passes Kubernetes runtime qualification; explicitly select inventory mode for the read-only preview") + } else { + store, err = artifacts.New(input, r.OutputPath, r.ExplicitOutput) + if err != nil { + fail(1, err.Error()) + } else { + progress.Info("[PASS] Configuration: %d hosts, %d DPUs; %s mode", len(input.Site.Discovery.Hosts), len(input.Site.Discovery.DPUs), resolved.Mode) + logger.V(1).Info("Discovery configuration ready", "mode", resolved.Mode, "sources", len(resolved.Targets), "timeout", resolved.Timeout, "requestTimeout", resolved.RequestTimeout, "concurrency", resolved.MaxConcurrentBMCs) + if !inv.Output.Quiet { + for _, endpoint := range resolved.Targets { + if endpoint.Access.InsecureSkipVerify() { + fmt.Fprintf(r.Stderr, "Warning: BMC TLS certificate verification disabled for %s\n", endpoint.Endpoint) + } + } + } + deps := discovery.Dependencies{SSH: sshprobe.Collector{}, BMC: redfish.Collector{Qualifications: catalog.Hardware()}, Probes: probe.Runner{Assets: assets, Journal: probe.FileJournal(filepath.Join(store.RunDir, "resources.json"))}} + if resolved.Input.Site.Inventory == nil { + resolved.Input.Site.Inventory = store.PriorInventory + } + report = discovery.Collect(ctx, resolved, store.RunID(), deps) + progress.Info("[RUNNING] Publication: saving inventory and detailed report") + publishStarted := time.Now() + publishCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second) + err = store.Publish(publishCtx, &report) + cancel() + if err != nil { + progress.Info("[FAIL] Publication incomplete; see the result for details") + report.Execution = "failed" + report.Errors = append(report.Errors, inventory.Problem{Code: "PUBLICATION_FAILED", Message: err.Error(), ExitCode: 1}) + _ = store.Report(&report) + } else { + progress.Info("[PASS] Publication: %s discovery artifacts saved", report.Discovery.Outcome) + } + logger.V(1).Info("Discovery publication finished", "outcome", report.Discovery.Outcome, "success", err == nil, "duration", time.Since(publishStarted)) + } + } + } + } + if inv.Output.Format == "json" { + if err := json.NewEncoder(r.Stdout).Encode(report); err != nil { + fmt.Fprintln(r.Stderr, "cannot encode DPF discovery result") + return apperrors.MarkReported(apperrors.NewGeneralError("cannot encode DPF discovery result", nil)) + } + } else { + if err := textReport(r.Stdout, report, input); err != nil { + fmt.Fprintln(r.Stderr, "cannot write DPF discovery result") + return apperrors.MarkReported(apperrors.NewGeneralError("cannot write DPF discovery result", nil)) + } + } + return apperrors.NewExitStatus(discovery.ExitCode(report)) + }), nil +} + +func validateManagement(r config.Request) error { + if r.Kubeconfig == "" { + return nil + } + _, err := discovery.ManagementAccess(r) + return err +} diff --git a/pkg/target/dpf/driver_test.go b/pkg/target/dpf/driver_test.go new file mode 100644 index 00000000..ec02ea13 --- /dev/null +++ b/pkg/target/dpf/driver_test.go @@ -0,0 +1,64 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package dpf + +import ( + "bytes" + "context" + "encoding/json" + "github.com/nvidia/k8s-launch-kit/pkg/errors" + "github.com/nvidia/k8s-launch-kit/pkg/target" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "testing" +) + +func TestDriver(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "DPF target") } + +var _ = Describe("target boundary", func() { + It("reports one JSON document for invalid input and returns exit two", func() { + var stdout, stderr bytes.Buffer + d := NewDiscoverAdapter(Request{Stdout: &stdout, Stderr: &stderr}) + op, e := d.Bind(target.Invocation{Target: target.DPF, Phase: target.Discover, Output: target.OutputOptions{Format: "json"}}) + Expect(e).NotTo(HaveOccurred()) + e = op.Run(context.Background()) + code, ok := errors.IsExitStatus(e) + Expect(ok).To(BeTrue()) + Expect(code).To(Equal(2)) + var result map[string]any + Expect(json.Unmarshal(stdout.Bytes(), &result)).To(Succeed()) + Expect(result["target"]).To(Equal("dpf")) + Expect(result["execution"]).To(Equal("failed")) + }) + It("advertises only discovery and rejects other phases before binding", func() { + for _, p := range []target.Phase{target.Generate, target.Deploy, target.Validate, target.Pipeline} { + _, e := NewDiscoverAdapter(Request{}).Bind(target.Invocation{Target: target.DPF, Phase: p}) + Expect(e).To(HaveOccurred()) + Expect(Descriptor().Capability(p).Available).To(BeFalse()) + } + }) + It("keeps live preparation status on stderr and suppresses it with quiet without changing JSON", func() { + for _, quiet := range []bool{false, true} { + var stdout, stderr bytes.Buffer + op, err := NewDiscoverAdapter(Request{Stdout: &stdout, Stderr: &stderr}).Bind(target.Invocation{Target: target.DPF, Phase: target.Discover, Output: target.OutputOptions{Format: "json", Quiet: quiet}}) + Expect(err).NotTo(HaveOccurred()) + Expect(errors.ExitCodeFromError(op.Run(context.Background()))).To(Equal(2)) + var result map[string]any + Expect(json.Unmarshal(stdout.Bytes(), &result)).To(Succeed()) + Expect(result["execution"]).To(Equal("failed")) + if quiet { + Expect(stderr.String()).To(BeEmpty()) + } else { + Expect(stderr.String()).To(ContainSubstring("[RUNNING] Configuration")) + Expect(stderr.String()).To(ContainSubstring("[FAIL] Discovery preparation")) + } + } + }) + It("returns a local failure when the human summary cannot be written", func() { + var stderr bytes.Buffer + op, err := NewDiscoverAdapter(Request{Stdout: failedWriter{}, Stderr: &stderr}).Bind(target.Invocation{Target: target.DPF, Phase: target.Discover, Output: target.OutputOptions{Format: "text", Quiet: true}}) + Expect(err).NotTo(HaveOccurred()) + Expect(errors.ExitCodeFromError(op.Run(context.Background()))).To(Equal(1)) + Expect(stderr.String()).To(ContainSubstring("cannot write DPF discovery result")) + }) +}) diff --git a/pkg/target/dpf/report.go b/pkg/target/dpf/report.go new file mode 100644 index 00000000..a4db67f8 --- /dev/null +++ b/pkg/target/dpf/report.go @@ -0,0 +1,284 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package dpf + +import ( + "fmt" + "io" + "slices" + "strings" + "text/tabwriter" + "unicode" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" +) + +// textReport presents the same decisions as the structured report. Internal +// record/evidence keys remain in JSON; they are never operator-facing names. +func textReport(w io.Writer, report inventory.Report, input config.Input) error { + var out strings.Builder + status := "incomplete" + switch { + case report.Execution == "cancelled": + status = "cancelled" + case report.Execution == "failed" || report.Discovery.Outcome == "invalid": + status = "failed" + case discovery.ExitCode(report) == 0: + status = "completed" + } + fmt.Fprintf(&out, "DPF discovery %s", status) + if report.Discovery.Mode != "" { + fmt.Fprintf(&out, " (%s)", displayText(report.Discovery.Mode)) + } + fmt.Fprintln(&out) + labels := reportLabels(input, report.Inventory) + passed, required := 0, 0 + for _, check := range report.Discovery.Checks { + if check.RequiredPhase == "discover" { + required++ + if check.State == "passed" { + passed++ + } + } + } + if required > 0 { + fmt.Fprintf(&out, "Required discovery checks: %d/%d passed\n", passed, required) + } + if inv := report.Inventory; inv != nil { + fmt.Fprintf(&out, "Identified: %d/%d hosts, %d/%d DPUs; host-to-DPU mappings: %d\n", observedCount(inv.Hosts), len(inv.Hosts), observedCount(inv.DPUs), len(inv.DPUs), len(inv.Associations)) + if inv.Software != nil { + fmt.Fprintf(&out, "Software comparison catalog: DPF %s\n", displayText(inv.Software.Version)) + } + writeDevices(&out, inv, report.Discovery.Checks, labels) + } + var blockers, later, deferred []inventory.Check + for _, check := range report.Discovery.Checks { + if check.State == "passed" { + continue + } + switch { + case check.RequiredPhase == "discover": + blockers = append(blockers, check) + case check.State == "not-requested" || check.State == "not-evaluated": + deferred = append(deferred, check) + default: + later = append(later, check) + } + } + if len(blockers)+len(report.Errors) > 0 || report.Discovery.Cleanup.State == "incomplete" { + fmt.Fprintln(&out, "\nProblems preventing successful discovery:") + writeChecks(&out, blockers, labels) + for _, problem := range report.Errors { + fmt.Fprintf(&out, " - %s: %s", subjectLabel(labels, problem.InputRef), displayText(problem.Message)) + if problem.Code != "" { + fmt.Fprintf(&out, " [%s]", displayText(problem.Code)) + } + if problem.Stage != "" { + fmt.Fprintf(&out, " (stage: %s)", displayText(problem.Stage)) + } + if problem.HTTPStatus != 0 { + fmt.Fprintf(&out, " (HTTP %d)", problem.HTTPStatus) + } + fmt.Fprintln(&out) + } + if report.Discovery.Cleanup.State == "incomplete" { + fmt.Fprintln(&out, " - Temporary resource/session cleanup is incomplete. Inspect the detailed report before retrying.") + } + } + if len(later) > 0 { + fmt.Fprintln(&out, "\nBefore provisioning (these do not fail inventory discovery):") + writeChecks(&out, later, labels) + } + if len(deferred) > 0 { + fmt.Fprintln(&out, "\nNot checked in this run:") + writeChecks(&out, deferred, labels) + } + if report.Inventory != nil { + fmt.Fprintf(&out, "\nCleanup: %s\n", displayText(report.Discovery.Cleanup.State)) + } + if len(report.Artifacts) > 0 { + fmt.Fprintln(&out, "\nSaved files:") + for _, artifact := range report.Artifacts { + name := map[string]string{"inventory": "Inventory", "partial": "Partial inventory", "report": "Detailed report", "source-backup": "Previous configuration"}[artifact.Kind] + if name == "" { + name = displayText(artifact.Kind) + } + fmt.Fprintf(&out, " %s: %s\n", name, displayText(artifact.Path)) + } + } + _, err := io.WriteString(w, out.String()) + return err +} + +func observedCount(devices []inventory.Device) int { + count := 0 + for _, device := range devices { + if device.ObservationState == "observed" && device.ID != "" { + count++ + } + } + return count +} + +func reportLabels(input config.Input, inv *inventory.Inventory) map[string]string { + labels := map[string]string{"management": "Management cluster", "/management": "Management cluster", "inventory": "Inventory", "": "Discovery"} + nameCounts := map[string]int{} + for _, endpoint := range append(slices.Clone(input.Site.Discovery.Hosts), input.Site.Discovery.DPUs...) { + if endpoint.Name != nil { + nameCounts[displayText(*endpoint.Name)]++ + } + } + for _, group := range []struct { + role string + endpoints []config.Endpoint + }{{"hosts", input.Site.Discovery.Hosts}, {"dpus", input.Site.Discovery.DPUs}} { + for i, endpoint := range group.endpoints { + label := endpoint.BMCEndpoint + if endpoint.SSH != nil { + label = endpoint.SSH.Address + } + if endpoint.Name != nil && *endpoint.Name != "" { + if nameCounts[displayText(*endpoint.Name)] == 1 { + label = *endpoint.Name + } else { + label = *endpoint.Name + " (" + label + ")" + } + } + labels[fmt.Sprintf("/discovery/%s/%d", group.role, i)] = displayText(label) + } + } + if inv != nil { + for _, device := range append(slices.Clone(inv.Hosts), inv.DPUs...) { + label := labels[device.InputRef] + if device.ReaderAdapter == "host-card-v1" && device.SerialNumber != "" { + label = "DPU " + device.SerialNumber + " (via " + subjectLabel(labels, device.InputRef) + ")" + } + if label == "" { + switch { + case device.BMCEndpoint != "": + label = device.BMCEndpoint + case device.SSHEndpoint != "": + label = device.SSHEndpoint + case device.SerialNumber != "": + label = "DPU " + device.SerialNumber + default: + label = "Unidentified source" + } + } + for _, key := range []string{device.RecordKey, device.ID, device.InputRef} { + if key == device.InputRef && device.ReaderAdapter == "host-card-v1" { + continue + } + if key != "" { + labels[key] = displayText(label) + } + } + } + } + return labels +} + +func subjectLabel(labels map[string]string, subject string) string { + if label, ok := labels[subject]; ok { + return label + } + return "Unidentified source" +} + +func writeDevices(out io.Writer, inv *inventory.Inventory, checks []inventory.Check, labels map[string]string) { + if len(inv.DPUs) == 0 { + return + } + hosts := map[string]string{} + for _, association := range inv.Associations { + hosts[association.DPUID] = subjectLabel(labels, association.HostID) + } + admin := map[string]bool{} + for _, check := range checks { + if check.ID == "dpf.management.bmc-administrator" { + admin[check.Subject] = check.State == "passed" + } + } + fmt.Fprintln(out) + table := tabwriter.NewWriter(out, 0, 4, 2, ' ', 0) + fmt.Fprintln(table, "DPU\tMODEL\tSERIAL\tHOST\tBMC ADMIN") + devices := slices.Clone(inv.DPUs) + slices.SortStableFunc(devices, func(a, b inventory.Device) int { + return strings.Compare(subjectLabel(labels, a.RecordKey), subjectLabel(labels, b.RecordKey)) + }) + for _, device := range devices { + host := hosts[device.ID] + if host == "" { + host = "not established" + } + administrator := "not verified" + if admin[device.RecordKey] { + administrator = "verified" + } + model, serial := device.Model, device.SerialNumber + if model == "" { + model = "unknown" + } + if serial == "" { + serial = "unknown" + } + fmt.Fprintf(table, "%s\t%s\t%s\t%s\t%s\n", subjectLabel(labels, device.RecordKey), displayText(model), displayText(serial), host, administrator) + } + _ = table.Flush() // The destination is the report's strings.Builder. +} + +func writeChecks(out io.Writer, checks []inventory.Check, labels map[string]string) { + // Coalesce identical findings while retaining all affected sources. Different + // messages (for example, different measured versions) remain separate. + type group struct { + check inventory.Check + subjects []string + } + groups := []group{} + index := map[[4]string]int{} + for _, check := range checks { + key := [4]string{check.ID, check.State, check.ReasonCode, check.Message} + i, found := index[key] + if !found { + i = len(groups) + index[key] = i + groups = append(groups, group{check: check}) + } + if !slices.Contains(groups[i].subjects, check.Subject) { + groups[i].subjects = append(groups[i].subjects, check.Subject) + } + } + for _, g := range groups { + for i, subject := range g.subjects { + g.subjects[i] = subjectLabel(labels, subject) + } + slices.Sort(g.subjects) + message := g.check.Message + switch g.check.ReasonCode { + case "QUALIFIED_ADAPTER_REQUIRED": + message = "Hardware/firmware qualification is not established in this build; this is not a detected hardware fault." + case "BOM_MODEL_UNSUPPORTED": + message = "Software compatibility is unknown: the selected discovery BOM has no matching model entry." + case "INVENTORY_MODE": + message = "Management-network connectivity probes were not requested in inventory mode." + case "DPF_AUTHENTICATION_NOT_EVALUATED": + message = "DPF controller authentication is checked during provisioning, not inventory discovery." + } + if message == "" { + message = "No passing result was established; inspect the detailed report." + } + fmt.Fprintf(out, " - %s: %s\n", strings.Join(g.subjects, ", "), displayText(message)) + } +} + +func displayText(value string) string { + return strings.Map(func(r rune) rune { + if unicode.IsControl(r) { + return ' ' + } + return r + }, value) +} diff --git a/pkg/target/dpf/report_test.go b/pkg/target/dpf/report_test.go new file mode 100644 index 00000000..aa9b4774 --- /dev/null +++ b/pkg/target/dpf/report_test.go @@ -0,0 +1,177 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +package dpf + +import ( + "bytes" + "encoding/json" + "errors" + "strings" + + "github.com/nvidia/k8s-launch-kit/pkg/dpf/config" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/discovery" + "github.com/nvidia/k8s-launch-kit/pkg/dpf/inventory" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func displayFixture() (inventory.Report, config.Input) { + input := config.Input{} + hostName, dpuName := "worker-a", "worker-a-dpu" + input.Site.Discovery.Hosts = []config.Endpoint{{Name: &hostName, BMCEndpoint: "https://192.0.2.21"}} + input.Site.Discovery.DPUs = []config.Endpoint{{Name: &dpuName, BMCEndpoint: "https://192.0.2.11"}} + report := discovery.NewReport() + report.Discovery.Outcome = "complete" + report.Discovery.Mode = "inventory" + report.Inventory = &inventory.Inventory{ + Hosts: []inventory.Device{{ID: "host-id", RecordKey: "sha256:host-record", InputRef: "/discovery/hosts/0", BMCEndpoint: "https://192.0.2.21", ObservationState: "observed"}}, + DPUs: []inventory.Device{{ID: "dpu-id", RecordKey: "sha256:dpu-record", InputRef: "/discovery/dpus/0", BMCEndpoint: "https://192.0.2.11", ObservationState: "observed", Model: "B4240", SerialNumber: "CARD-A"}}, + Associations: []inventory.Association{{DPUID: "dpu-id", HostID: "host-id", SerialNumber: "CARD-A"}}, + } + report.Discovery.Checks = []inventory.Check{ + {ID: "dpf.management.bmc-administrator", Subject: "sha256:dpu-record", RequiredPhase: "discover", State: "passed", ReasonCode: "BMC_ADMIN_VERIFIED"}, + {ID: "dpf.inventory.association", Subject: "sha256:dpu-record", RequiredPhase: "discover", State: "passed"}, + {ID: "dpf.device.qualification", Subject: "sha256:host-record", RequiredPhase: "generate", State: "failed", ReasonCode: "QUALIFIED_ADAPTER_REQUIRED"}, + {ID: "dpf.device.qualification", Subject: "sha256:dpu-record", RequiredPhase: "generate", State: "failed", ReasonCode: "QUALIFIED_ADAPTER_REQUIRED"}, + {ID: "dpf.software.applicability", Subject: "sha256:dpu-record", RequiredPhase: "generate", State: "unknown", ReasonCode: "BOM_MODEL_UNSUPPORTED"}, + {ID: "dpf.management.path", Subject: "management", RequiredPhase: "generate", State: "not-requested", ReasonCode: "INVENTORY_MODE"}, + {ID: "dpf.management.dpf-authentication", Subject: "sha256:dpu-record", RequiredPhase: "deploy", State: "not-evaluated", ReasonCode: "DPF_AUTHENTICATION_NOT_EVALUATED"}, + } + report.Artifacts = []inventory.Artifact{{Kind: "inventory", Path: "/results/discovered.yaml"}, {Kind: "report", Path: "/results/report.json"}} + return report, input +} + +var _ = Describe("human DPF discovery report", func() { + It("shows named mappings and separates discovery success from later-stage gaps without mutating evidence", func() { + report, input := displayFixture() + before, err := json.Marshal(report) + Expect(err).NotTo(HaveOccurred()) + var out bytes.Buffer + Expect(textReport(&out, report, input)).To(Succeed()) + text := out.String() + Expect(text).To(ContainSubstring("DPF discovery completed (inventory)")) + Expect(text).To(ContainSubstring("Required discovery checks: 2/2 passed")) + Expect(text).To(ContainSubstring("Identified: 1/1 hosts, 1/1 DPUs; host-to-DPU mappings: 1")) + Expect(text).To(ContainSubstring("worker-a-dpu")) + Expect(text).To(ContainSubstring("CARD-A")) + Expect(text).To(ContainSubstring("B4240")) + Expect(text).To(ContainSubstring("verified")) + Expect(text).To(ContainSubstring("Before provisioning (these do not fail inventory discovery):")) + Expect(strings.Count(text, "Hardware/firmware qualification")).To(Equal(1)) + Expect(text).To(ContainSubstring("worker-a, worker-a-dpu:")) + Expect(text).To(ContainSubstring("Not checked in this run:")) + Expect(text).To(ContainSubstring("Software compatibility is unknown")) + Expect(text).To(ContainSubstring("Detailed report: /results/report.json")) + Expect(text).NotTo(ContainSubstring("sha256:")) + Expect(text).NotTo(ContainSubstring("dpf.device.qualification")) + Expect(text).NotTo(ContainSubstring("Problems preventing")) + after, err := json.Marshal(report) + Expect(err).NotTo(HaveOccurred()) + Expect(after).To(Equal(before)) + }) + + It("keeps a required administrator failure blocking and names the affected DPU", func() { + report, input := displayFixture() + report.Discovery.Outcome = "partial" + report.Discovery.Checks[0].State = "failed" + report.Discovery.Checks[0].ReasonCode = "BMC_ADMIN_REQUIRED" + report.Discovery.Checks[0].Message = "The BMC account lacks Administrator privileges." + report.Artifacts = []inventory.Artifact{{Kind: "partial", Path: "/run/partial.yaml"}} + var out bytes.Buffer + Expect(textReport(&out, report, input)).To(Succeed()) + Expect(out.String()).To(ContainSubstring("DPF discovery incomplete")) + Expect(out.String()).To(ContainSubstring("Required discovery checks: 1/2 passed")) + Expect(out.String()).To(ContainSubstring("Problems preventing successful discovery:")) + Expect(out.String()).To(ContainSubstring("worker-a-dpu: The BMC account lacks Administrator privileges.")) + Expect(out.String()).To(ContainSubstring("not verified")) + Expect(out.String()).To(ContainSubstring("Partial inventory: /run/partial.yaml")) + Expect(discovery.ExitCode(report)).To(Equal(2)) + }) + + It("does not claim overall success after a publication failure or cancellation", func() { + for _, state := range []string{"failed", "cancelled"} { + report, input := displayFixture() + report.Execution = state + report.Errors = []inventory.Problem{{Code: "PUBLICATION_FAILED", Message: "output changed during collection", ExitCode: 1}} + var out bytes.Buffer + Expect(textReport(&out, report, input)).To(Succeed()) + Expect(out.String()).To(ContainSubstring("DPF discovery " + state)) + Expect(out.String()).NotTo(ContainSubstring("DPF discovery completed")) + Expect(out.String()).To(ContainSubstring("output changed during collection [PUBLICATION_FAILED]")) + Expect(out.String()).To(ContainSubstring("Inventory: /results/discovered.yaml")) + } + }) + + It("handles invalid configuration, cleanup failure and unknown subjects without exposing hashes", func() { + report := discovery.NewReport() + report.Execution = "failed" + report.Discovery.Cleanup.State = "incomplete" + report.Errors = []inventory.Problem{{Code: "INVALID_CONFIGURATION", Message: "password reference is missing", ExitCode: 2}} + report.Discovery.Checks = []inventory.Check{{Subject: "sha256:unknown", RequiredPhase: "discover", State: "unknown", Message: "Source could not be identified."}} + var out bytes.Buffer + Expect(textReport(&out, report, config.Input{})).To(Succeed()) + Expect(out.String()).To(ContainSubstring("Unidentified source: Source could not be identified.")) + Expect(out.String()).To(ContainSubstring("cleanup is incomplete")) + Expect(out.String()).NotTo(ContainSubstring("sha256:")) + }) + + It("uses endpoints when names are absent and distinguishes duplicate names", func() { + report, input := displayFixture() + input.Site.Discovery.DPUs[0].Name = nil + labels := reportLabels(input, report.Inventory) + Expect(labels["sha256:dpu-record"]).To(Equal("https://192.0.2.11")) + input.Site.Discovery.DPUs[0].Name = input.Site.Discovery.Hosts[0].Name + labels = reportLabels(input, report.Inventory) + Expect(labels["sha256:dpu-record"]).To(Equal("worker-a (https://192.0.2.11)")) + Expect(labels["sha256:host-record"]).To(Equal("worker-a (https://192.0.2.21)")) + }) + + It("distinguishes names that collide after terminal sanitation and retains all source identities", func() { + report, input := displayFixture() + hostName, dpuName := "node\nx", "node\tx" + input.Site.Discovery.Hosts[0].Name = &hostName + input.Site.Discovery.DPUs[0].Name = &dpuName + labels := reportLabels(input, report.Inventory) + Expect(labels["sha256:host-record"]).To(Equal("node x (https://192.0.2.21)")) + Expect(labels["sha256:dpu-record"]).To(Equal("node x (https://192.0.2.11)")) + Expect(labels["/management"]).To(Equal("Management cluster")) + // Even unknown subjects with the same fallback label remain two sources. + var out bytes.Buffer + writeChecks(&out, []inventory.Check{{Subject: "one", Message: "Unavailable"}, {Subject: "two", Message: "Unavailable"}}, labels) + Expect(out.String()).To(ContainSubstring("Unidentified source, Unidentified source: Unavailable")) + }) + + It("does not let a host-derived DPU replace the host's display identity", func() { + report, input := displayFixture() + report.Inventory.DPUs[0].ReaderAdapter = "host-card-v1" + report.Inventory.DPUs[0].InputRef = "/discovery/hosts/0" + labels := reportLabels(input, report.Inventory) + Expect(labels["/discovery/hosts/0"]).To(Equal("worker-a")) + Expect(labels["sha256:dpu-record"]).To(Equal("DPU CARD-A (via worker-a)")) + }) + + It("keeps different version findings and removes terminal control characters", func() { + report, input := displayFixture() + report.Inventory.DPUs[0].Model = "B4240\n\x1b[31m" + report.Discovery.Checks = []inventory.Check{ + {ID: "dpf.software.nic", State: "failed", RequiredPhase: "generate", Subject: "sha256:dpu-record", Message: "NIC version A is below minimum."}, + {ID: "dpf.software.nic", State: "failed", RequiredPhase: "generate", Subject: "sha256:dpu-record", Message: "NIC version B is below minimum."}, + } + var out bytes.Buffer + Expect(textReport(&out, report, input)).To(Succeed()) + Expect(out.String()).To(ContainSubstring("NIC version A")) + Expect(out.String()).To(ContainSubstring("NIC version B")) + Expect(out.String()).NotTo(ContainSubstring("\x1b")) + Expect(out.String()).NotTo(ContainSubstring("B4240\n")) + }) + + It("returns the output writer's failure", func() { + report, input := displayFixture() + Expect(textReport(failedWriter{}, report, input)).To(MatchError("write failed")) + }) +}) + +type failedWriter struct{} + +func (failedWriter) Write([]byte) (int, error) { return 0, errors.New("write failed") } diff --git a/pkg/target/dpf/signals_unix.go b/pkg/target/dpf/signals_unix.go new file mode 100644 index 00000000..d8b24489 --- /dev/null +++ b/pkg/target/dpf/signals_unix.go @@ -0,0 +1,12 @@ +//go:build !windows + +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package dpf + +import ( + "os" + "syscall" +) + +func terminationSignals() []os.Signal { return []os.Signal{os.Interrupt, syscall.SIGTERM} } diff --git a/pkg/target/dpf/signals_windows.go b/pkg/target/dpf/signals_windows.go new file mode 100644 index 00000000..e6e508ee --- /dev/null +++ b/pkg/target/dpf/signals_windows.go @@ -0,0 +1,9 @@ +//go:build windows + +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 +package dpf + +import "os" + +func terminationSignals() []os.Signal { return []os.Signal{os.Interrupt} } diff --git a/pkg/yamlinput/node.go b/pkg/yamlinput/node.go new file mode 100644 index 00000000..641ff7c4 --- /dev/null +++ b/pkg/yamlinput/node.go @@ -0,0 +1,108 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// SPDX-License-Identifier: Apache-2.0 + +// Package yamlinput provides target-neutral source-preserving YAML operations. +package yamlinput + +import ( + "bytes" + "fmt" + yaml3 "gopkg.in/yaml.v3" +) + +func Expand(node *yaml3.Node) (*yaml3.Node, error) { + budget := 200000 + return expand(node, map[*yaml3.Node]bool{}, &budget) +} +func expand(node *yaml3.Node, active map[*yaml3.Node]bool, budget *int) (*yaml3.Node, error) { + *budget-- + if *budget < 0 { + return nil, fmt.Errorf("expanded YAML exceeds node limit") + } + if node == nil { + return nil, fmt.Errorf("missing YAML node") + } + if active[node] { + return nil, fmt.Errorf("cyclic YAML alias") + } + active[node] = true + defer delete(active, node) + if node.Kind == yaml3.AliasNode { + return expand(node.Alias, active, budget) + } + copy := *node + copy.Anchor = "" + copy.Content = nil + for _, child := range node.Content { + expanded, err := expand(child, active, budget) + if err != nil { + return nil, err + } + copy.Content = append(copy.Content, expanded) + } + if copy.Kind != yaml3.MappingNode { + return ©, nil + } + entries := copy.Content + copy.Content = nil + for i := 0; i+1 < len(entries); i += 2 { + if entries[i].Tag != "!!merge" { + continue + } + mappings := []*yaml3.Node{entries[i+1]} + if entries[i+1].Kind == yaml3.SequenceNode { + mappings = entries[i+1].Content + } + for _, mapping := range mappings { + if mapping.Kind != yaml3.MappingNode { + return nil, fmt.Errorf("YAML merge value must be a mapping or sequence of mappings") + } + for j := 0; j+1 < len(mapping.Content); j += 2 { + putYAMLField(©, mapping.Content[j], mapping.Content[j+1], false) + } + } + } + for i := 0; i+1 < len(entries); i += 2 { + if entries[i].Tag != "!!merge" { + putYAMLField(©, entries[i], entries[i+1], true) + } + } + return ©, nil +} + +func putYAMLField(mapping, key, value *yaml3.Node, replace bool) { + for i := 0; i+1 < len(mapping.Content); i += 2 { + if mapping.Content[i].Value == key.Value { + if replace { + mapping.Content[i], mapping.Content[i+1] = key, value + } + return + } + } + mapping.Content = append(mapping.Content, key, value) +} + +func Encode(node *yaml3.Node) ([]byte, error) { + var b bytes.Buffer + e := yaml3.NewEncoder(&b) + e.SetIndent(2) + if err := e.Encode(node); err != nil { + return nil, err + } + if err := e.Close(); err != nil { + return nil, err + } + return b.Bytes(), nil +} +func Set(mapping *yaml3.Node, key string, value *yaml3.Node) { + for i := 0; i < len(mapping.Content); i += 2 { + if mapping.Content[i].Value == key { + value.HeadComment = mapping.Content[i+1].HeadComment + value.LineComment = mapping.Content[i+1].LineComment + value.FootComment = mapping.Content[i+1].FootComment + mapping.Content[i+1] = value + return + } + } + mapping.Content = append(mapping.Content, &yaml3.Node{Kind: yaml3.ScalarNode, Tag: "!!str", Value: key}, value) +} diff --git a/scripts/check-docs.py b/scripts/check-docs.py index 41ad620d..a8e4309f 100644 --- a/scripts/check-docs.py +++ b/scripts/check-docs.py @@ -5,7 +5,7 @@ Only --help and explicitly constructed offline generate commands are executed. Documentation shell text is parsed, never executed. -Architecture plans describe future interfaces and are excluded from syntax checks. +Engineering plans and API internals live outside the published docs tree. """ import argparse @@ -141,7 +141,7 @@ def check_index(): route = "/k8s-launch-kit/" + ("" if path == "index.md" else path[:-3] + "/") if route not in index: raise ValueError(f"llms.txt is missing {route}") - print(f"Navigation: all {len(paths)} published pages, including linked plans, are listed in llms.txt") + print(f"Navigation: all {len(paths)} published pages are listed in llms.txt") def check_offline_examples(binary): diff --git a/scripts/generate-workflow-docs.py b/scripts/generate-workflow-docs.py new file mode 100644 index 00000000..df1e4831 --- /dev/null +++ b/scripts/generate-workflow-docs.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +# Copyright 2026 NVIDIA CORPORATION & AFFILIATES +# SPDX-License-Identifier: Apache-2.0 +"""Render repository-owned OpenSpec procedure metadata; never execute commands.""" + +import argparse +import html +import json +from pathlib import Path +import re +import sys + +ROOT = Path(__file__).resolve().parents[1] +SOURCE = ROOT / "openspec/workflows/dpf-discovery.json" +OUTPUT = ROOT / "docs/user/dpf-workflows.md" +ID = re.compile(r"^[a-z][a-z0-9-]*$") + + +def text(value, where): + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{where} must be nonempty text") + + +def keys(value, required, optional, where): + if not isinstance(value, dict): + raise ValueError(f"{where} must be an object") + if set(value) - required - optional or required - set(value): + raise ValueError(f"{where} has unknown or missing fields") + + +def validate(doc): + keys(doc, {"formatVersion", "title", "introduction", "workflows"}, {"$comment"}, "document") + if type(doc["formatVersion"]) is not int or doc["formatVersion"] != 1: + raise ValueError("unsupported formatVersion") + text(doc["title"], "title") + text(doc["introduction"], "introduction") + if not isinstance(doc["workflows"], list) or not doc["workflows"]: + raise ValueError("workflows must be a nonempty list") + workflow_ids = set() + for workflow in doc["workflows"]: + keys(workflow, {"id", "title", "introduction", "steps"}, set(), "workflow") + identifier = workflow["id"] + if not isinstance(identifier, str) or not ID.fullmatch(identifier) or identifier in workflow_ids: + raise ValueError("invalid or duplicate workflow ID") + workflow_ids.add(identifier) + text(workflow["title"], "workflow title") + text(workflow["introduction"], "workflow introduction") + if not isinstance(workflow["steps"], list) or not workflow["steps"]: + raise ValueError("steps must be a nonempty list") + step_ids = set() + for step in workflow["steps"]: + keys(step, {"id", "title", "body", "transitions"}, set(), "step") + identifier = step["id"] + if not isinstance(identifier, str) or not ID.fullmatch(identifier) or identifier in step_ids: + raise ValueError("invalid or duplicate step ID") + step_ids.add(identifier) + text(step["title"], "step title") + text(step["body"], "step body") + if not isinstance(step["transitions"], list): + raise ValueError("transitions must be a list") + for step in workflow["steps"]: + destinations = set() + for transition in step["transitions"]: + keys(transition, {"to", "label"}, set(), "transition") + if not isinstance(transition["to"], str) or transition["to"] not in step_ids: + raise ValueError("transition points to an unknown step") + text(transition["label"], "transition label") + if transition["to"] in destinations: + raise ValueError("duplicate transition destination") + destinations.add(transition["to"]) + + +def diagram_text(value): + # Titles/labels are data, not Mermaid markup. In particular, newlines and + # quotes cannot terminate a diagram node or inject another graph statement. + return html.escape(" ".join(value.split()), quote=True) + + +def render(doc): + validate(doc) + lines = ["", "", f"# {doc['title']}", "", doc["introduction"].strip(), ""] + for workflow in doc["workflows"]: + lines += [f"## {workflow['title']} {{#{workflow['id']}}}", "", + workflow["introduction"].strip(), "", "```mermaid", "flowchart TD"] + # Sequential node identifiers avoid Mermaid keyword collisions (e.g. end). + nodes = {step["id"]: f"n{i}" for i, step in enumerate(workflow["steps"], 1)} + for step in workflow["steps"]: + lines.append(f' {nodes[step["id"]]}["{diagram_text(step["title"])}"]') + for step in workflow["steps"]: + for transition in step["transitions"]: + lines.append(f' {nodes[step["id"]]} -->|"{diagram_text(transition["label"])}"| {nodes[transition["to"]]}') + lines += ["```", ""] + for number, step in enumerate(workflow["steps"], 1): + lines += [f"### {number}. {step['title']} {{#{workflow['id']}-{step['id']}}}", + "", step["body"].strip(), ""] + if step["transitions"]: + links = [f"[{t['label']}](#{workflow['id']}-{t['to']})" for t in step["transitions"]] + lines += ["Next: " + "; ".join(links) + ".", ""] + return "\n".join(lines) + + +def unique_keys(pairs): + result = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON field: {key}") + result[key] = value + return result + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--check", action="store_true", help="fail on drift without writing output") + parser.add_argument("--source", type=Path, default=SOURCE) + parser.add_argument("--output", type=Path, default=OUTPUT) + args = parser.parse_args(argv) + try: + content = render(json.loads(args.source.read_text(encoding="utf-8"), object_pairs_hook=unique_keys)) + if args.check: + if not args.output.is_file() or args.output.read_text(encoding="utf-8") != content: + raise ValueError("workflow documentation is missing or stale; run scripts/generate-workflow-docs.py") + else: + args.output.write_text(content, encoding="utf-8") + except (OSError, ValueError) as error: + print(f"Workflow documentation: {error}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/test_generate_workflow_docs.py b/scripts/test_generate_workflow_docs.py new file mode 100644 index 00000000..d33e61a8 --- /dev/null +++ b/scripts/test_generate_workflow_docs.py @@ -0,0 +1,84 @@ +# Copyright 2026 NVIDIA CORPORATION & AFFILIATES +# SPDX-License-Identifier: Apache-2.0 +"""Documentation generation contracts; no CLI or infrastructure is invoked.""" + +import copy +from contextlib import redirect_stderr +import importlib.util +import io +import json +from pathlib import Path +import tempfile +import unittest + +SPEC = importlib.util.spec_from_file_location("workflow_docs", Path(__file__).with_name("generate-workflow-docs.py")) +generator = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(generator) + + +class WorkflowDocsTests(unittest.TestCase): + def setUp(self): + self.doc = {"formatVersion": 1, "title": "Procedures", "introduction": "Read first.", "workflows": [ + {"id": "inventory", "title": "Inventory", "introduction": "Collect facts.", "steps": [ + {"id": "run", "title": "Run", "body": "```sh\nprintf 'example only'\n```", "transitions": [{"to": "inspect", "label": "complete"}]}, + {"id": "inspect", "title": "Inspect", "body": "Read the report.", "transitions": []}]}]} + + def test_deterministic_procedure_and_graph(self): + original = copy.deepcopy(self.doc) + rendered = generator.render(self.doc) + self.assertEqual(rendered, generator.render(self.doc)) + self.assertEqual(self.doc, original) + self.assertIn('n1 -->|"complete"| n2', rendered) + self.assertIn("[complete](#inventory-inspect)", rendered) + self.assertIn("printf 'example only'", rendered) + self.assertIn("### 2. Inspect", rendered) + + def test_rejects_duplicate_ids_and_unknown_destinations(self): + for change in ( + lambda d: d["workflows"].append(copy.deepcopy(d["workflows"][0])), + lambda d: d["workflows"][0]["steps"].append(copy.deepcopy(d["workflows"][0]["steps"][0])), + lambda d: d["workflows"][0]["steps"][0]["transitions"][0].update(to="absent"), + ): + doc = copy.deepcopy(self.doc) + change(doc) + with self.assertRaises(ValueError): + generator.render(doc) + + def test_rejects_malformed_shapes_and_unknown_fields(self): + for change in ( + lambda d: d.update(formatVersion=True), + lambda d: d.update(workflows=[]), + lambda d: d.update(command="must not execute"), + lambda d: d["workflows"][0].update(steps=[]), + lambda d: d["workflows"][0]["steps"][0].update(transitions="bad"), + ): + doc = copy.deepcopy(self.doc) + change(doc) + with self.assertRaises(ValueError): + generator.render(doc) + with self.assertRaises(ValueError): + json.loads('{"id":"one", "id":"two"}', object_pairs_hook=generator.unique_keys) + + def test_escapes_diagram_labels(self): + self.doc["workflows"][0]["steps"][0]["title"] = 'Read "report"\n