Add k8s-network SP QE test plan and deploy provider for FLPATH-4865

[vkolodny]

Document manual E2E smoke for the FLPATH-3227 network provider epic, with layer ownership separation and utilities deploy wiring on host port 8090.

[qodo-code-review]

PR Summary by Qodo

Add Kubernetes network provider deployment and QE smoke plan

✨ Enhancement 📝 Documentation ⚙️ Configuration changes 🕐 10-20 Minutes

AI Description

• Registers the Kubernetes network provider with the utilities deployment framework.
• Adds temporary Compose wiring exposing the provider on host port 8090.
• Documents manual smoke coverage, ownership boundaries, prerequisites, and future automation.

Diagram

graph TD
  User["QE Operator"] --> Deploy["Deploy Script"] --> Registry["Provider Config"] --> Compose["Compose Override"] --> SP["Network Provider"] --> Cluster["Kubernetes Cluster"]
  SP --> Control["DCM Control Plane"]
  SP --> NATS["NATS"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Use an upstream Compose profile

• ➕ Avoids duplicating the provider service definition in utilities
• ➕ Keeps image, dependency, and environment configuration owned by control-plane
• ➖ The required control-plane profile does not exist yet
• ➖ Would block current QE deployment and manual smoke validation
• ➖ A small override may still be required to publish host port 8090

Recommendation: Use the PR's registry entry and temporary full Compose override now because they follow the existing provider extension mechanism without modifying deploy-dcm.sh. Replace the full override with the upstream control-plane profile once available, retaining only utilities-specific port wiring if necessary.

Files changed (3) +372 / -0

Documentation (1) +318 / -0

FLPATH-3227-k8s-network-sp.mdDefine the manual Kubernetes network provider QE plan +318/-0

Define the manual Kubernetes network provider QE plan

• Documents phased smoke scenarios for deployment, registration, catalog discovery, CRUD, load balancer behavior, catalog provisioning, and API validation. It also separates utilities E2E ownership from provider unit and integration coverage and records automation prerequisites and exit criteria.

test-plans/FLPATH-3227-k8s-network-sp.md

Other (2) +54 / -0

k8s-network.confRegister the Kubernetes network service provider +25/-0

Register the Kubernetes network service provider

• Adds provider-registry metadata for the deployment flag, Compose override, namespace, kubeconfig export, CLI requirement, and cluster namespace validation. This makes the provider available through explicit and all-provider deployments.

providers/k8s-network.conf

compose-k8s-network-sp.yamlAdd temporary Compose deployment for the network provider +29/-0

Add temporary Compose deployment for the network provider

• Defines the provider image, DCM and NATS connectivity, namespace and kubeconfig settings, dependencies, and read-only cluster credentials. The provider API is published on host port 8090 until control-plane supplies a native Compose profile.

tests/compose-k8s-network-sp.yaml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Versioned deploys run a main image ✓ Resolved 🐞 Bug ≡ Correctness

Description

The new compose service tags its image with K8S_NETWORK_SERVICE_PROVIDER_VERSION, but that
variable is absent from VERSION_ENV_VARS. When deployment uses --version or --version release,
every existing DCM image is pinned while the network provider silently retains its main default.

Code

tests/compose-k8s-network-sp.yaml[9]

+    image: quay.io/dcm-project/k8s-network-service-provider:${K8S_NETWORK_SERVICE_PROVIDER_VERSION:-main}

Evidence

The new image defaults to main through its provider-specific variable, while the deployment script
exports DCM_VERSION only through an explicit list that does not contain that variable.

tests/compose-k8s-network-sp.yaml[9-10]
scripts/deploy-dcm.sh[28-36]
scripts/deploy-dcm.sh[1047-1063]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Global version pinning does not set the image version variable introduced for the Kubernetes network provider, causing versioned deployments to use its `main` image.

## Fix Focus Areas
- scripts/deploy-dcm.sh[28-36]
- tests/compose-k8s-network-sp.yaml[9-9]

## Recommended Fix
Add `K8S_NETWORK_SERVICE_PROVIDER_VERSION` to `VERSION_ENV_VARS` so `--version` and release resolution export the same version for the new provider as for all other DCM images.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Catalog smoke cannot select an agent ✓ Resolved 🐞 Bug ≡ Correctness

Description

E2E-07 jumps from catalog-item creation to instance creation without discovering the network agent
or creating a global routing policy that selects it. When no suitable pre-existing policy happens to
be installed, placement cannot deterministically route the instance to the provider whose behavior
this case is meant to verify.

Code

test-plans/FLPATH-3227-k8s-network-sp.md[R238-240]

+| 1 | Create CatalogItem for `network` type | Success |
+| 2 | Create CatalogItemInstance | Placement routes to network SP |
+| 3 | `oc get svc -n <ns>` | Service with DCM labels |

Evidence

The repository's implemented catalog flow discovers an agent and creates a GLOBAL policy selecting
it before creating the catalog-item instance, whereas E2E-07 omits both prerequisites while
expecting placement to route to the new provider.

test-plans/FLPATH-3227-k8s-network-sp.md[234-241]
tests/e2e/core_platform_test.go[63-69]
tests/e2e/core_platform_test.go[130-154]
tests/e2e/core_platform_test.go[156-175]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The catalog-instance smoke expects placement to select the network provider without setting up the routing policy required by the repository's corresponding catalog E2E flow.

## Fix Focus Areas
- test-plans/FLPATH-3227-k8s-network-sp.md[232-241]
- tests/e2e/core_platform_test.go[63-69]
- tests/e2e/core_platform_test.go[130-175]

## Recommended Fix
Add steps to discover the network agent, create a temporary global routing policy whose result selects that agent, verify the resulting instance assignment, and remove the policy during cleanup.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Shell checks cannot identify the dialect ✓ Resolved 📘 Rule violation ✧ Quality

Description

providers/k8s-network.conf begins with a descriptive comment even though load_providers executes
the file with source. Because it is a sourced Bash library rather than an executable, it supplies
neither a shebang nor a # shellcheck shell=bash directive identifying its intended shell.

Code

providers/k8s-network.conf[1]

+# Kubernetes Network Service Provider

Evidence

Compliance rule 3176515 requires sourced Bash library files to declare # shellcheck shell=bash
instead of an executable shebang. The provider configuration lacks that declaration, and the
deployment script confirms that provider configurations are loaded using source.

Rule 3176515: Bash scripts must use #!/usr/bin/env bash shebang
providers/k8s-network.conf[1-8]
scripts/deploy-dcm.sh[64-76]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new provider configuration is sourced as Bash code but does not contain the required ShellCheck dialect directive for sourced library files.

## Fix Focus Areas
- providers/k8s-network.conf[1-1]

## Recommended Fix
Add `# shellcheck shell=bash` at the beginning of the configuration file before its descriptive comments.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Upstream cluster smoke fails without oc ✓ Resolved 🐞 Bug ≡ Correctness

Description

The @lab-default smoke steps invoke oc directly even though the provider accepts upstream
Kubernetes environments with only kubectl. On such a deployment, service verification and cleanup
stop with a missing-command error despite the deployment script having accepted the environment.

Code

test-plans/FLPATH-3227-k8s-network-sp.md[198]

+| 3 | `oc get svc e2e-clusterip-smoke -n <ns>` | `TYPE=ClusterIP`, DCM labels present |

Evidence

The new provider explicitly declares an oc-or-kubectl requirement, and the repository's E2E helper
searches for either executable, but the manual plan uses only oc for several mandatory steps.

providers/k8s-network.conf[15-15]
test-plans/FLPATH-3227-k8s-network-sp.md[96-105]
test-plans/FLPATH-3227-k8s-network-sp.md[198-224]
test-plans/FLPATH-3227-k8s-network-sp.md[267-273]
tests/e2e/sp_helpers_test.go[293-316]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The test plan advertises support for any Kubernetes cluster but hard-codes the OpenShift-only `oc` executable for verification and cleanup.

## Fix Focus Areas
- test-plans/FLPATH-3227-k8s-network-sp.md[94-106]
- test-plans/FLPATH-3227-k8s-network-sp.md[194-224]
- test-plans/FLPATH-3227-k8s-network-sp.md[263-273]

## Recommended Fix
Define a cluster CLI variable that selects available `oc` or `kubectl`, then use that variable for every service inspection and cleanup command in the plan.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Deploy guidance omits the new provider ✓ Resolved 📘 Rule violation ⚙ Maintainability

Description

providers/k8s-network.conf adds the generated --k8s-network-service-provider and
--k8s-network-namespace options plus provider-specific environment and compose wiring without
changing either README.md or CLAUDE.md. The existing deploy guidance and Cursor deploy prompt
enumerate the older provider set, prerequisites, environment overrides, and ports, so developers and
assistant-generated commands do not discover this deployment path.

Code

providers/k8s-network.conf[R8-10]

+PROVIDER_LABEL="k8s-network"
+PROVIDER_FLAG="k8s-network-service-provider"
+PROVIDER_DESCRIPTION="Enable the k8s network service provider"

Evidence

Compliance rule 2901048 requires README.md or CLAUDE.md to reflect new deployment behavior and
requires affected Cursor guidance to be updated. The new registry entry creates deployment and
namespace flags, while README.md, CLAUDE.md, and the Cursor deploy prompt still list only the
previously supported providers and ports.

Rule 2901048: Update project documentation and cursor configs when behavior or conventions change
providers/k8s-network.conf[8-18]
README.md[46-85]
CLAUDE.md[45-82]
.cursor/prompts/deploy-dcm.md[5-111]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The provider registry introduces new deployment flags, environment variables, prerequisites, compose behavior, and host port 8090, but the primary project and Cursor deployment guidance still documents only the older provider set.

## Fix Focus Areas
- README.md[46-85]
- CLAUDE.md[45-82]
- .cursor/prompts/deploy-dcm.md[5-111]

## Recommended Fix
Add the Kubernetes network provider to the deployment prerequisites and examples, document its namespace and kubeconfig environment variables and host port, update the current-provider list, and add equivalent guidance to the Cursor deployment prompt.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 18 rules

Review mode: ⚖️ Balanced: This adds runtime deployment configuration and a compose service with Kubernetes credentials, networking, registration, and port wiring, alongside a large operational test plan, so it carries meaningful integration and configuration risk but is not dense enough to warrant extended review.

Tip of the day

💡 Did you know, you can ask Qodo to dismiss a finding you disagree with, with your reason on record

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[vkolodny]

Qodo review — addressed in 8824405

1. High — K8S_NETWORK_SERVICE_PROVIDER_VERSION missing from VERSION_ENV_VARS

Fixed. Added K8S_NETWORK_SERVICE_PROVIDER_VERSION to the VERSION_ENV_VARS array in scripts/deploy-dcm.sh so --version / DCM_VERSION pinning applies to the network SP image tag in tests/compose-k8s-network-sp.yaml.

2. High — E2E-07 missing routing policy + agent discovery

Fixed. Expanded E2E-07 in test-plans/FLPATH-3227-k8s-network-sp.md to mirror core_platform_test.go: discover network agent → create catalog item → create GLOBAL routing policy with Rego selected_agent → create catalog item instance → poll for RUNNING → verify agent_name and $CLUSTER_CLI get svc. Included example catalog item and policy JSON payloads.

3. Medium — README / CLAUDE / deploy-dcm prompt not documenting network provider

Fixed.

• README.md: deploy example (FLPATH-3542 Automate Container SP E2E Tests #6), DCM_NETWORK_SP_URL (port 8090), run-e2e.sh flag example
• CLAUDE.md: added k8s-network to provider list, port 8090, compose-override note, K8S_NETWORK_SERVICE_PROVIDER_VERSION
• .cursor/prompts/deploy-dcm.md: prerequisite, deploy commands, K8S_NETWORK_SP_NAMESPACE / version env vars

4. Medium — k8s-network.conf shellcheck directive

Fixed. Added # shellcheck shell=bash at top of providers/k8s-network.conf (same pattern as sourced scripts/kind/kind-env.sh).

5. Medium — test plan hard-codes oc

Fixed. Prerequisites and cleanup now define CLUSTER_CLI (oc or kubectl). All verification steps (E2E-04, E2E-05, E2E-06, E2E-07, cleanup) use $CLUSTER_CLI instead of hard-coded oc.

[jordigilh]

curious: we don't support HTTPS yet? at this point I'd expect all endpoints to have TLS supported. Is it a known gap or is it just a missed configuration setting?

[jordigilh]

why not validate protocol and port as well?

[jordigilh]

you don't check that the port is available first to ensure no env pollution?

[jordigilh]

what's holding us back from investing in this automation today? It feels like a waste to define a test plan that explicitly requires manual execution.

[vkolodny]

Re: port availability (E2E-01) — @jordigilh

We don't have automated port-free checks in utilities E2E or deploy-dcm.sh.

Closest precedent is the auth test plan prerequisite list (ports must be free before deploy); on Jenkins CI we use the 9080 remap workaround (FLPATH-4421) instead of a pre-check.

I'll add an explicit prerequisite to E2E-01: ports 8080 and 8090 must be free before deploy (or document the 9080 override + DCM_GATEWAY_URL when 8080 is occupied), to match that pattern.

[jordigilh]

Good catch. E2E here is manual QE smoke for now (not CI yet),

What's holding you back from implementing the CI lane since you already have the tests here? It feels like a waste not to automate this when you have already the bulk of the work done.

[vkolodny]

Re: HTTPS — @jordigilh

HTTP is intentional for this smoke path.

Local compose (deploy-dcm.sh + utilities overrides) exposes control-plane and SP APIs on plain HTTP — same as container/ACM SP E2E. The stack isn't set up for TLS termination in that layout (SP_SERVER_ADDRESS: ":8080", http:// endpoints in compose). Production registration URLs can be https://; that's the advertised endpoint, not how we run lab smoke.

TLS for the control-plane API is tracked in the auth epic. Cert management for network services is a v1 non-goal in the enhancement.

Added a one-liner in the port map so that's explicit in the test plan.

[vkolodny]

Re: protocol/port validation — @jordigilh

Fair point. Step 4 only checked kubernetes.type — I'll extend it to assert spec.ports[0] round-trips on GET (protocol: TCP, port: 80, target_port: 8080).

Full port validation matrix stays in the SP repo; this is just smoke fidelity.

[vkolodny]

Re: protocol/port validation — @jordigilh

Fair point. Step 4 only checked kubernetes.type — I'll extend it to assert spec.ports[0] round-trips on GET (protocol: TCP, port: 80, target_port: 8080).

Full port validation matrix stays in the SP repo; this is just smoke fidelity.

[vkolodny]

Re: automation — @jordigilh

Manual plan first — same approach as container (checklist now, Ginkgo spec later). This PR also adds the first utilities deploy path for the network SP.

CRUD is implemented in k8s-network-service-provider#2 but not merged to main yet — I verified main still returns 500 on POST/GET/DELETE. Hard to automate E2E-04+ until that lands (and the Quay image is published).

E2E-01–03 could be automated in a follow-up; full suite after #2 merges.

[vkolodny]

@jordigilh — the test plan documents the cases; there’s no Ginkgo implementation yet (network_sp_*_test.go doesn’t exist). That’s the CI lane gap.

Same blockers as above: SP PR #2 not on main (CRUD still 500 there), Quay image not published. E2E-01–03 could go into utilities CI in a follow-up; E2E-04+ after #2 merges.

[jordigilh]

Re: automation — @jordigilh

Manual plan first — same approach as container (checklist now, Ginkgo spec later). This PR also adds the first utilities deploy path for the network SP.

CRUD is implemented in k8s-network-service-provider#2 but not merged to main yet — I verified main still returns 500 on POST/GET/DELETE. Hard to automate E2E-04+ until that lands (and the Quay image is published).

So what are you manually testing then if main is failing outright?

E2E-01–03 could be automated in a follow-up; full suite after #2 merges.

Is there any jira issue to capture the e2e work that you are not implementing here?

I fail to see the value of manual testing as a deliverable, to be honest.

[vkolodny]

@jordigilh — fair pushback, clarifying:

What manual testing covers today (no CRUD needed): E2E-01–03 — deploy + compose wiring, agent registration (/agents), catalog network service type. That validates this PR’s deliverable (utilities deploy path on 8090). E2E-04+ need CRUD and are blocked until SP #2 is on main + Quay image exists.

Automation: agreed it should land in utilities CI — planned as network_sp_*_test.go (documented in the test plan Future automation). No separate Jira yet; I can open a follow-up under FLPATH-3227 for the Ginkgo/CI work once #2 merges.

Why manual for FLPATH-4865: epic QE smoke sign-off while SP CRUD is still landing — same pattern we used for container before sp_container_api_test.go existed. Not a substitute for CI; the checklist becomes the automation spec.

[gpb88]

Is there a reason we are only testing ClusterIP and Loadbalancer resources?
What about NodePort? (routing_level omitted + node_ports present)
To get full coverage we are also missing:

• Loadbalancer with node_ports specified -> validate ports on the LB
• routing_level: application in the spec -> validate errors

This would cover all possible cases

[vkolodny]

@jordigilh — agreed, automation only. Updated the test plan on the branch to reflect that — no manual execution as the deliverable.

[vkolodny]

@gpb88 — fair point. Extended the plan with E2E-09 (NodePort), E2E-10 (LB + node_ports), and E2E-11 (routing_level: application error) so we cover the full v1 inference matrix once at utilities E2E.

[tkiss28]

lgtm