diff --git a/.claude/skills/nvcf-self-hosted-local-dev b/.claude/skills/nvcf-self-hosted-local-dev new file mode 120000 index 000000000..33bbe2b32 --- /dev/null +++ b/.claude/skills/nvcf-self-hosted-local-dev @@ -0,0 +1 @@ +../../ai-tooling/dev/skills/nvcf-self-hosted-local-dev \ No newline at end of file diff --git a/.codex/skills/nvcf-self-hosted-local-dev b/.codex/skills/nvcf-self-hosted-local-dev new file mode 120000 index 000000000..33bbe2b32 --- /dev/null +++ b/.codex/skills/nvcf-self-hosted-local-dev @@ -0,0 +1 @@ +../../ai-tooling/dev/skills/nvcf-self-hosted-local-dev \ No newline at end of file diff --git a/.cursor/skills/nvcf-self-hosted-local-dev b/.cursor/skills/nvcf-self-hosted-local-dev new file mode 120000 index 000000000..33bbe2b32 --- /dev/null +++ b/.cursor/skills/nvcf-self-hosted-local-dev @@ -0,0 +1 @@ +../../ai-tooling/dev/skills/nvcf-self-hosted-local-dev \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index bba043414..62c04ad12 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -71,6 +71,17 @@ secrets, or artifact directories without explicit user confirmation. If creating a net-new environment, use unique cluster names, ports, Helmfile environments, secrets files, CLI configs, and artifact directories. +Use separate control-plane and compute-plane clusters for validation that +affects worker registration, callbacks, request routing, reverse tunnels, +transport PKI, DNS, or cross-cluster endpoints. A single-cluster result is +supplemental and does not prove cross-cluster behavior. + +Light split-cluster validation is sufficient unless broader coverage is +requested. Launch a real worker on the compute cluster and complete at least +one end-to-end invocation through the control plane. If the invocation fails, +capture evidence from both clusters and identify the first broken hop. Use +`.cursor/skills/nvcf-self-hosted-local-dev/SKILL.md` for the reusable workflow. + ## GitLab CI Manual Actions Treat manual GitLab CI actions as remote write operations. Before triggering a @@ -214,6 +225,7 @@ public snapshot; still follow OSS Snapshot Hygiene manually before finishing. |-------|----------|---------| | `documentation-style` | `ai-tooling/dev/skills/` | Public docs, AGENTS, and skill-writing style | | `nvcf-explore-stack` | `ai-tooling/dev/skills/` | Navigate the self-hosted stack topology and dependency graph | +| `nvcf-self-hosted-local-dev` | `ai-tooling/dev/skills/` | Select and validate local single-cluster or split-cluster self-hosted topology | | `nvca-chart-release` | `ai-tooling/dev/skills/` | Release NVCA Operator chart changes from monorepo source to the vendored Helm chart | | `nvca-self-managed-install` | `ai-tooling/dev/skills/` | Install or validate the NVCA Operator chart against a self-managed control plane | | `nvca-values-customization` | `ai-tooling/dev/skills/` | Customize NVCA Operator Helm chart values in the monorepo | diff --git a/ai-tooling/README.md b/ai-tooling/README.md index f3d19517f..37df73935 100644 --- a/ai-tooling/README.md +++ b/ai-tooling/README.md @@ -8,6 +8,7 @@ Public agent skills for users and developers working with NVIDIA Cloud Functions |-------|-------------| | [documentation-style](dev/skills/documentation-style/SKILL.md) | Public docs, AGENTS, and skill-writing style | | [nvcf-explore-stack](dev/skills/nvcf-explore-stack/SKILL.md) | Navigate the self-hosted stack topology, helmfile dependency graph, chart ownership, and deployment order | +| [nvcf-self-hosted-local-dev](dev/skills/nvcf-self-hosted-local-dev/SKILL.md) | Select and validate local single-cluster or split-cluster self-hosted topology with real worker traffic | | [nvcf-self-managed-installation](user/skills/nvcf-self-managed-installation/SKILL.md) | Install and deploy the nvcf-self-managed-stack helmfile bundle: installation, teardown, values overrides, pull secrets, debugging | | [nvcf-self-managed-cli](user/skills/nvcf-self-managed-cli/SKILL.md) | Standalone NVCF CLI (`nvcf-cli`) for self-managed/self-hosted deployments: install, status, add compute plane, teardown, function lifecycle, invocation, and API keys | | [nvcf-self-managed-prerequisite](user/skills/nvcf-self-managed-prerequisite/SKILL.md) | Install the cluster-level prerequisites NVCA needs: KAI Scheduler (with queue-quota patch) and the SMB CSI driver. Cloud-neutral helm installs pinned to NVCF-validated versions | diff --git a/ai-tooling/dev/skills/nvcf-self-hosted-local-dev/SKILL.md b/ai-tooling/dev/skills/nvcf-self-hosted-local-dev/SKILL.md new file mode 100644 index 000000000..b2600dd1f --- /dev/null +++ b/ai-tooling/dev/skills/nvcf-self-hosted-local-dev/SKILL.md @@ -0,0 +1,75 @@ +--- +name: nvcf-self-hosted-local-dev +description: >- + Plan and validate local NVCF self-hosted k3d environments with + topology-aware single-cluster or split-cluster coverage. Use for local QA, + BDD, worker connectivity, cross-cluster endpoints, request routing, reverse + tunnels, transport PKI, DNS, or real function invocation. +license: Apache-2.0 +compatibility: Requires a local NVCF checkout, Docker, k3d, kubectl, Helm, and access to required NVCF artifacts +author: "nvcf-core-eng " +version: "1.0.0" +tags: [nvcf, self-hosted, k3d, local-development, testing] +tools: [Read, Shell] +metadata: + internal: false + author: "nvcf-core-eng " + version: "1.0.0" + tags: [nvcf, self-hosted, k3d, local-development, testing] + languages: [bash, yaml] + frameworks: [k3d, kubernetes, helmfile] + domain: cloud-infrastructure +--- + +# NVCF Self-Hosted Local Development + +## Instructions + +Use the smallest topology that can prove the claim. Use one cluster for basic +installation, rendering, and function lifecycle checks that do not cross a +cluster boundary. Use separate control-plane and compute-plane clusters for +worker registration, callbacks, request routing, reverse tunnels, transport +PKI, DNS, or cross-cluster endpoint changes. A single-cluster pass is +supplemental for those paths. + +Read the [local development guide](../../../../docs/dev/local-development.md) +before creating clusters. For CLI-driven split topology, also read the +[multi-cluster CLI flow](../../../../docs/user/local-development/multi-cluster-cli.md). + +## Prepare an isolated environment + +1. Start from a fresh worktree at the target ref. +2. Inventory existing clusters, contexts, ports, tools, and local artifacts. +3. Ask before deleting shared state. Prefer fresh task-owned clusters. +4. Use unique cluster names, ports, kubeconfigs, Helmfile environments, CLI + configs, secrets files, and evidence directories. +5. Record the target commit and tool versions before installation. + +## Validate current behavior + +Test the target code before applying aliases, routes, insecure transport, or +other workarounds. For a topology-sensitive claim: + +1. Install the control plane in one cluster and register a separate compute + cluster. +2. Probe worker-facing DNS names and ports from the compute cluster. A + control-plane `*.svc.cluster.local` name does not cross cluster boundaries + unless the topology creates a compute-cluster alias or route for it. +3. Launch a real worker in the compute cluster. +4. Send at least one end-to-end invocation through the control plane and + validate the response. + +One successful invocation is enough for light verification. Installation, +registration, ready pods, or rendered values alone do not prove worker +traffic. + +## Capture failure evidence + +If the invocation fails, identify the first broken hop. Capture bounded pod +logs, events, endpoint and DNS probes, transport trust state, and relevant +resource summaries from both clusters. Do not capture credentials, tokens, or +private keys. + +Keep failed clusters running when debugging is requested. Report cluster +names, kubeconfig paths, exposed endpoints, evidence locations, and exact +reattachment commands. Do not clean up until the user approves it. diff --git a/tests/bdd/AGENTS.md b/tests/bdd/AGENTS.md index 274ccbd41..eb0ea73ec 100644 --- a/tests/bdd/AGENTS.md +++ b/tests/bdd/AGENTS.md @@ -175,6 +175,20 @@ multi-cluster feature: ... no matching key(s) found"). Switch the context to the compute cluster BEFORE `make register-cluster`, not after. +## Topology-sensitive coverage + +Multi-cluster installation and registration do not prove worker traffic. +Changes to worker endpoints, callbacks, request routers, reverse tunnels, +transport PKI, or DNS need a split-cluster check that launches a real worker on +the compute cluster and invokes it through the control plane. Profile and +render assertions and single-cluster invocations are supplemental. + +Probe worker-facing names and ports from the compute cluster. Do not rely on a +control-plane `*.svc.cluster.local` name crossing the cluster boundary. If the +topology creates a compute-cluster alias Service and Endpoints for that name, +probe the alias before invoking. On failure, collect evidence from both +clusters and identify the first broken hop. + ## Tests - Every Go file under `tests/bdd/` carries the SPDX header in diff --git a/tools/ncp-local-cluster/AGENTS.md b/tools/ncp-local-cluster/AGENTS.md index efce1c983..b5e145639 100644 --- a/tools/ncp-local-cluster/AGENTS.md +++ b/tools/ncp-local-cluster/AGENTS.md @@ -23,6 +23,19 @@ Cluster lifecycle targets require local tools such as `k3d`, `kubectl`, `helm`, For detailed local k3d workflow and cleanup safety, see `docs/dev/local-development.md` from the repo root. +## Split-cluster validation + +Use `build-and-deploy-multicluster` for behavior that crosses control and +compute planes. Cluster creation, compute registration, and NVCA readiness do +not prove worker traffic. Probe worker-facing DNS and ports from a pod in the +compute cluster, then launch a real worker and invoke it through the control +plane. + +Do not rely on a control-plane `*.svc.cluster.local` name crossing the cluster +boundary. A compute-cluster alias or external route must be part of the +topology under test and must be probed from the compute cluster. Record the +baseline and first broken hop before applying a workaround. + ## Ownership This subtree is monorepo-native. Do not sync it from the old standalone repo.