feat(deploy): add compose and in-cluster Kind deployment

[jenniferubah]

Summary

• Add standalone compose stack (deploy/compose.yaml) for NATS + environment-agent with Kind helpers pointing to utilities scripts
• Add in-cluster Kind deployment (deploy/k8s/) with make k8s-deploy, NodePort access, and k8s-host-urls.sh for verify/publish
• Add sample create requests (deploy/samples/, make publish-creates / make k8s-publish-creates) and deployment docs (DEPLOY.md, compose-kind.md, in-cluster.md)

Type of Change

• New feature (deployment)

Stacked PR Status

• Yes, this is part of a stack:
  • Parent PR: #48 (Base PR - Open for review)

Assisted-By: Cursor AI

[qodo-code-review]

PR Summary by Qodo

Add Compose and in-cluster Kind deployment workflows

✨ Enhancement 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Adds Compose and in-cluster Kind stacks for NATS and the environment agent.
• Automates networking, KubeVirt installation, deployment, verification, and sample request
 publishing.
• Documents both deployment models, configuration, RBAC, troubleshooting, and teardown.

Diagram

graph TD
  OP["Developer"] --> MAKE["Make targets"]
  MAKE --> COMPOSE["Compose stack"] --> AGENT["Environment agent"] --> API["Kind API"] --> WORK["SP workloads"]
  MAKE --> K8S["Kubernetes manifests"] --> AGENT
  COMPOSE --> NATS["NATS JetStream"] --> AGENT
  K8S --> NATS

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Helm-based in-cluster deployment

• ➕ Provides reusable values, release lifecycle management, and easier environment-specific configuration.
• ➕ Could align deployment packaging with the control-plane Helm stack.
• ➖ Adds chart maintenance and templating overhead for a local Kind workflow.
• ➖ May duplicate or couple this repository to control-plane deployment concerns.

2. kubectl port-forward host access

• ➕ Works consistently across Kind runtimes without relying on NodePort routing.
• ➕ Avoids fixed cluster-wide NodePort assignments.
• ➖ Requires persistent background processes for verification and publishing.
• ➖ Complicates automation and endpoint discovery across multiple services.

Recommendation: The Compose plus Kustomize approach is appropriate for repository-local development because it remains explicit, lightweight, and easy to inspect. Retain fixed NodePorts for the current one-command workflow, but consider optional port-forward support if Docker Desktop or Podman networking proves inconsistent.

Files changed (26) +1350 / -3

Enhancement (1) +2 / -1

ContainerfilePrepare writable agent persistence directory +2/-1

Prepare writable agent persistence directory

• Creates '/app/data' before ownership and permission changes so the non-root agent can persist embedded SP registrations.

Containerfile

Documentation (4) +432 / -0

README.mdLink local Compose and in-cluster deployment guides +19/-0

Link local Compose and in-cluster deployment guides

• Adds a concise Kind and Compose quick start and links to the detailed deployment models.

README.md

DEPLOY.mdDocument supported local deployment models +116/-0

Document supported local deployment models

• Introduces quick starts, configuration guidance, sample publishing, teardown, and script references for Compose and in-cluster deployments.

deploy/DEPLOY.md

compose-kind.mdExplain Compose-to-Kind connectivity +82/-0

Explain Compose-to-Kind connectivity

• Documents kubeconfig rewriting, shared runtime networking, KubeVirt prerequisites, service exposure, teardown, and troubleshooting.

deploy/docs/compose-kind.md

in-cluster.mdGuide in-cluster agent deployment +215/-0

Guide in-cluster agent deployment

• Documents Kind deployment, in-cluster authentication, workload RBAC, persistence, verification, sample requests, and troubleshooting.

deploy/docs/in-cluster.md

Other (21) +916 / -2

.gitignoreIgnore local deployment secrets and generated kubeconfig +4/-0

Ignore local deployment secrets and generated kubeconfig

• Excludes the deployment environment file and Compose-specific Kind kubeconfig from version control.

.gitignore

MakefileAdd Compose and Kind deployment automation +57/-2

Add Compose and Kind deployment automation

• Adds targets for Compose lifecycle, Kind networking, KubeVirt, in-cluster deployment, verification, and sample publishing. It also validates container-engine discovery and explicitly builds from 'Containerfile'.

Makefile

.env.exampleProvide local deployment configuration template +53/-0

Provide local deployment configuration template

• Documents environment overrides for agent identity, DCM connectivity, embedded providers, persistence, and image selection.

deploy/.env.example

compose.yamlDefine standalone NATS and environment-agent stack +75/-0

Define standalone NATS and environment-agent stack

• Adds a Compose stack with JetStream persistence, one-shot stream initialization, agent configuration, host control-plane access, and Kind kubeconfig mounting.

deploy/compose.yaml

environment-agent.yamlDeploy and expose the in-cluster environment agent +70/-0

Deploy and expose the in-cluster environment agent

• Defines the agent Deployment with embedded container and VM providers, ServiceAccount authentication, registration storage, and a fixed NodePort service.

deploy/k8s/environment-agent.yaml

kustomization.yamlAssemble in-cluster deployment resources +7/-0

Assemble in-cluster deployment resources

• Collects namespace, NATS, RBAC, and environment-agent manifests into one Kustomize application.

deploy/k8s/kustomization.yaml

namespace.yamlCreate dedicated DCM namespace +4/-0

Create dedicated DCM namespace

• Defines the namespace used by the local in-cluster NATS and agent resources.

deploy/k8s/namespace.yaml

nats-init-job.yamlInitialize required JetStream streams +26/-0

Initialize required JetStream streams

• Adds an idempotent Job that creates the control-plane-owned request, status, and response streams after NATS starts.

deploy/k8s/nats-init-job.yaml

nats.yamlDeploy NATS JetStream with NodePort access +47/-0

Deploy NATS JetStream with NodePort access

• Defines a single-node NATS deployment with ephemeral JetStream storage and host-reachable client access.

deploy/k8s/nats.yaml

rbac.yamlGrant embedded providers workload permissions +36/-0

Grant embedded providers workload permissions

• Creates the agent ServiceAccount and binds namespaced permissions for container and KubeVirt workload management in 'default'.

deploy/k8s/rbac.yaml

container-create-spec.jsonAdd sample container create specification +19/-0

Add sample container create specification

• Provides an nginx container request with representative CPU and memory constraints for local routing checks.

deploy/samples/container-create-spec.json

vm-create-spec.jsonAdd sample VM create specification +23/-0

Add sample VM create specification

• Provides a Fedora VM request with CPU, memory, and boot disk settings for KubeVirt routing checks.

deploy/samples/vm-create-spec.json

install-kubevirt.shAutomate KubeVirt installation on Kind +18/-0

Automate KubeVirt installation on Kind

• Applies a configurable KubeVirt release to the active Kind context and waits for availability.

deploy/scripts/install-kubevirt.sh

k8s-deploy.shAutomate image build and in-cluster rollout +55/-0

Automate image build and in-cluster rollout

• Builds and optionally loads the agent image into Kind, applies manifests, initializes JetStream, and waits for service readiness.

deploy/scripts/k8s-deploy.sh

k8s-host-urls.shResolve host-reachable in-cluster endpoints +81/-0

Resolve host-reachable in-cluster endpoints

• Detects whether Kind NodePorts are reachable through localhost or the node IP and exports matching agent and NATS URLs.

deploy/scripts/k8s-host-urls.sh

kind-connect.shAttach Kind to the Compose network +23/-0

Attach Kind to the Compose network

• Connects the active Kind control-plane container to the Compose network with a certificate-compatible Kubernetes alias.

deploy/scripts/kind-connect.sh

kind-disconnect.shDetach Kind before Compose teardown +43/-0

Detach Kind before Compose teardown

• Finds the Kind container runtime and safely disconnects the node from known Compose networks before network removal.

deploy/scripts/kind-disconnect.sh

kind-env.shCentralize Kind context and runtime detection +79/-0

Centralize Kind context and runtime detection

• Provides shared helpers to derive the Kind node from kubectl context and select a runtime that sees both the node and Compose network.

deploy/scripts/kind-env.sh

kubeconfig-for-compose.shGenerate a container-reachable Kind kubeconfig +43/-0

Generate a container-reachable Kind kubeconfig

• Flattens the active Kind kubeconfig and rewrites its API endpoint to the Compose network alias, with ownership and path safeguards.

deploy/scripts/kubeconfig-for-compose.sh

publish-create-requests.shPublish sample create CloudEvents +114/-0

Publish sample create CloudEvents

• Builds container and VM create events from sample specs and publishes them through the NATS CLI or a containerized fallback.

deploy/scripts/publish-create-requests.sh

verify.shVerify agent health and embedded providers +39/-0

Verify agent health and embedded providers

• Checks the configured agent endpoint, reports container diagnostics on failure, and prints health and provider responses.

deploy/scripts/verify.sh

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Init uses PID variables ✓ Resolved 🐞 Bug ≡ Correctness

Description

The Kubernetes Job runs $$1, $$2, and $$3 directly under /bin/sh, where $$ expands to the
shell PID rather than escaping Compose interpolation. It therefore looks up or creates streams with
values such as <pid>1 instead of the requested stream names, and k8s-deploy.sh fails while
waiting for this Job.

Code

deploy/k8s/nats-init-job.yaml[R18-21]

+                nats --server nats://nats:4222 stream info "$$1" >/dev/null 2>&1 && return 0
+                echo "Creating JetStream stream $$1 (subjects: $$2)"
+                nats --server nats://nats:4222 stream add "$$1" \
+                  --subjects "$$2" --storage file --retention "$$3" \

Evidence

The Job invokes /bin/sh -ec directly, while its function references use $$N; the deployment
script applies this manifest directly and waits for completion. The sibling Compose script needs
doubled dollars only because Compose performs an extra interpolation pass.

deploy/k8s/nats-init-job.yaml[14-26]
deploy/scripts/k8s-deploy.sh[39-42]
deploy/compose.yaml[38-50]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The directly applied Kubernetes Job uses Compose-style dollar escaping, causing `/bin/sh` to expand the PID instead of function positional parameters.

## Issue Context
Unlike `deploy/compose.yaml`, this manifest is not interpolated by Compose. Use `$1`, `$2`, and `$3` in the Job script.

## Fix Focus Areas
- deploy/k8s/nats-init-job.yaml[17-22]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Unconditional container-engine check breaks all make targets ✓ Resolved 🐞 Bug ≡ Correctness

Description

The new top-level ifeq ($(CONTAINER_ENGINE),) / $(error ...) block is evaluated at Makefile
parse time for every invocation, so environments without Podman or Docker cannot run
container-independent targets such as build, test, lint, fmt, vet, or generate-*.
Container availability should not be a global prerequisite because the existing scoped
check-container-engine target already validates it for image-build.

Code

Makefile[R11-13]

+ifeq ($(CONTAINER_ENGINE),)
+$(error No supported container engine found. Please install podman or docker, or set CONTAINER_ENGINE explicitly.)
+endif

Evidence

The conditional block is at the top level outside any recipe, so GNU Make evaluates $(error ...)
while parsing the Makefile, before selecting the requested target. The Makefile defines many
non-container targets that invoke only Go tooling, while image-build already depends on the
dedicated check-container-engine target further down, demonstrating that runtime validation can be
limited to commands that actually require a container engine.

Makefile[11-13]
Makefile[213-220]
Makefile[3-13]
Makefile[33-37]
Makefile[81-90]
Makefile[217-220]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description

The top-level `ifeq ($(CONTAINER_ENGINE),)` / `$(error ...)` block is evaluated unconditionally at parse time, preventing every Make target from running on machines without Docker or Podman, including Go-only development and CI targets such as `build`, `test`, `lint`, `fmt`, `vet`, and `generate-*`.

## Issue Context

Container detection should remain lazy and apply only to commands that require a container runtime. The Makefile already defines a scoped `check-container-engine` target used as a prerequisite for `image-build`; retain that approach or use an equivalent target-specific check rather than making container availability a global prerequisite.

## Fix Focus Areas

- Makefile[3-13]
- Makefile[213-220]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. NATS init has no readiness wait in compose ✓ Resolved 🐞 Bug ☼ Reliability

Description

nats-init depends on nats only through condition: service_started, which means the container
process has launched rather than NATS being ready on port 4222, while the service has no healthcheck
and the initializer performs no retry or wait around its CLI calls. On slower systems, set -ec can
terminate this one-shot container after connection failures, and because restart: "no" is set and
environment-agent requires service_completed_successfully, initialization and the entire Compose
stack can remain blocked or fail to start.

Code

deploy/compose.yaml[R32-50]

+  nats-init:
+    image: docker.io/natsio/nats-box:0.14.3
+    depends_on:
+      nats:
+        condition: service_started
+    restart: "no"
+    entrypoint: ["/bin/sh", "-ec"]
+    command:
+      - |
+        ensure() {
+          nats --server nats://nats:4222 stream info "$$1" >/dev/null 2>&1 && return 0
+          echo "Creating JetStream stream $$1 (subjects: $$2)"
+          nats --server nats://nats:4222 stream add "$$1" \
+            --subjects "$$2" --storage file --retention "$$3" \
+            --discard old --max-age 72h --defaults
+        }
+        ensure dcm-agent-requests 'dcm.agent.>' limits
+        ensure dcm-status 'dcm.*' limits
+        ensure dcm-agent-responses dcm.agents.responses work

Evidence

The nats service has no healthcheck, and nats-init waits only for service_started, which
guarantees container startup but not that NATS is accepting connections. Its ensure() function
makes a single stream info attempt followed by stream add; if both fail to connect during the
normal startup window, /bin/sh -ec exits the initializer, which does not restart, while
environment-agent explicitly waits for it to complete successfully.

deploy/compose.yaml[23-50]
deploy/compose.yaml[65-69]
deploy/compose.yaml[23-37]
deploy/compose.yaml[38-50]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`nats-init` in `deploy/compose.yaml` can fail intermittently because it starts when the `nats` container process launches, before NATS is necessarily listening on port 4222, and its `ensure()` function does not retry connection or stream-creation failures. Because the initializer uses `set -ec`, has `restart: "no"`, and `environment-agent` depends on its successful completion, this race can break `make compose-up` and leave the agent blocked.

## Issue Context
The Compose configuration gates initialization only on `condition: service_started` and defines no NATS healthcheck, so it does not establish actual service readiness. Add a NATS healthcheck and depend on `service_healthy`, or retry readiness and stream creation inside `nats-init` before giving up; unlike Compose, the Kubernetes path explicitly waits for `deployment/nats` to become available before running the equivalent initialization Job and also uses `backoffLimit` retries as a safety net.

## Fix Focus Areas
- deploy/compose.yaml[23-37]
- deploy/compose.yaml[41-50]
- deploy/compose.yaml[65-69]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. kind-disconnect hardcoded network list ignores custom project name ✓ Resolved 🐞 Bug ≡ Correctness

Description

kind-disconnect.sh's fallback network list (`environment-agent_default control-plane_default
deploy_default) is hardcoded and unrelated to COMPOSE_PROJECT_NAME`, unlike the Makefile's
COMPOSE_NETWORK which is derived from it. If a user overrides COMPOSE_PROJECT_NAME (e.g. `make
compose-down COMPOSE_PROJECT_NAME=myproject), the actual network myproject_default` is never
disconnected because it's absent from the script's fallback list.

Code

deploy/scripts/kind-disconnect.sh[12]

+NETWORKS="${COMPOSE_NETWORKS:-environment-agent_default control-plane_default deploy_default}"

Evidence

Makefile derives COMPOSE_NETWORK from COMPOSE_PROJECT_NAME (Makefile:16-17) and exports
COMPOSE_PROJECT_NAME (Makefile:23), but kind-disconnect.sh's NETWORKS fallback at line 12 is a
separate hardcoded list unrelated to COMPOSE_PROJECT_NAME, so a custom project name's network is not
included.

deploy/scripts/kind-disconnect.sh[12]
Makefile[15-17]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
deploy/scripts/kind-disconnect.sh's NETWORKS fallback list is hardcoded and does not account for a custom COMPOSE_PROJECT_NAME, so overriding the project name leaves the actual compose network never disconnected.

## Fix Focus Areas
- deploy/scripts/kind-disconnect.sh[12]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Default Kind NodePorts unreachable ✓ Resolved 🐞 Bug ≡ Correctness

Description

The documented kind create cluster --name dcm-local creates no host port mappings, yet
verification relies exclusively on localhost or the node container's InternalIP. On Docker Desktop
hosts the container-network InternalIP is not host-routable and localhost has no mapped NodePort, so
make k8s-verify and make k8s-publish-creates cannot reach this deployment.

Code

deploy/scripts/k8s-host-urls.sh[R20-27]

+	# Docker Desktop Kind sometimes exposes NodePorts on localhost.
+	if agent_healthy "http://127.0.0.1:${AGENT_NODE_PORT}"; then
+		echo "http://127.0.0.1:${AGENT_NODE_PORT}"
+		return 0
+	fi
+
+	node_ip="$(node_internal_ip)"
+	if [[ -n "${node_ip}" ]] && agent_healthy "http://${node_ip}:${AGENT_NODE_PORT}"; then

Evidence

Both services expose fixed NodePorts, and the resolver tests only localhost and the first node's
InternalIP. The supplied cluster creation command has no Kind extraPortMappings, so the localhost
branch has no configured mapping and the workflow has no fallback such as port-forwarding.

deploy/scripts/k8s-host-urls.sh[17-32]
deploy/k8s/environment-agent.yaml[62-70]
deploy/k8s/nats.yaml[39-47]
deploy/docs/in-cluster.md[26-31]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The default Kind creation instructions do not expose the fixed NodePorts to the host, while URL resolution assumes either localhost exposure or direct routing to the Kind node container.

## Issue Context
Provide and document a Kind config with `extraPortMappings` for ports 30081 and 30422, or make the helper establish portable access such as `kubectl port-forward`.

## Fix Focus Areas
- deploy/scripts/k8s-host-urls.sh[17-32]
- deploy/k8s/environment-agent.yaml[62-70]
- deploy/k8s/nats.yaml[39-47]
- deploy/docs/in-cluster.md[26-31]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. In-cluster manifest uses unreachable host.docker.internal ✓ Resolved 🐞 Bug ≡ Correctness

Description

deploy/k8s/environment-agent.yaml hardcodes
DCM_REGISTRATION_URL=http://host.docker.internal:8080, but pods in a Kind cluster do not get this
hostname resolved (it is only wired via compose's extra_hosts, absent here), and no
dnsPolicy/hostAliases/CoreDNS customization is added by this PR. This causes make k8s-deploy to
ship a Deployment whose registration will fail to resolve DNS out of the box, contradicting the
documented in-cluster.md guidance to use a Kubernetes service name like dcm-control-plane.

Code

deploy/k8s/environment-agent.yaml[R35-37]

+            # Required; agent retries until reachable. Point at host control-plane when running one.
+            - name: DCM_REGISTRATION_URL
+              value: http://host.docker.internal:8080

Evidence

host.docker.internal is a Docker/compose-specific alias (only configured via extra_hosts in
deploy/compose.yaml); Kind pods use ClusterFirst DNS via CoreDNS and have no equivalent mapping. No
k8s-deploy.sh, kind-env.sh, or kustomization patch adds this DNS entry for the in-cluster path,
while docs (in-cluster.md) recommend a real service name instead.

deploy/k8s/environment-agent.yaml[35-37]
deploy/docs/in-cluster.md[88-91]
deploy/compose.yaml[70-73]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`deploy/k8s/environment-agent.yaml` sets DCM_REGISTRATION_URL to http://host.docker.internal:8080, which does not resolve inside Kind pods by default.

## Issue Context
This is the sample in-cluster manifest applied by `make k8s-deploy`; it's meant to work out of the box per `in-cluster.md`.

## Fix Focus Areas
- deploy/k8s/environment-agent.yaml[35-37]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View medium (2)

7. pull_policy always defeats local build ✓ Resolved 🐞 Bug ☼ Reliability

Description

The environment-agent service in deploy/compose.yaml sets pull_policy: always while also
declaring a build: section, so Compose attempts to pull
quay.io/dcm-project/environment-agent:main from the registry even when make compose-up passes
--build. This can break first-time/offline contributors if the tag doesn't exist or isn't
reachable, or silently use the pulled image instead of the freshly built local one.

Code

deploy/compose.yaml[R56-57]

+    image: quay.io/dcm-project/environment-agent:${ENVIRONMENT_AGENT_VERSION:-main}
+    pull_policy: always

Evidence

Compose applies pull_policy for services with both image and build; always pulls regardless of
--build. The file's own comment says local dev builds from the Containerfile, contradicting the
always-pull policy.

deploy/compose.yaml[52-57]
Makefile[40-41]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
deploy/compose.yaml sets pull_policy: always on a service that also has a build: section used for local development via `make compose-up --build`, which can cause compose to pull a possibly nonexistent/inaccessible remote image instead of building locally.

## Fix Focus Areas
- deploy/compose.yaml[52-57]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

8. Registration store is ephemeral ✓ Resolved 🐞 Bug ☼ Reliability

Description

Compose places the provider registration file in /app/data but mounts no volume there, so any
container recreation—including the advertised compose-up --build workflow—discards registered
provider state. This defeats the application's file-backed persistence for the standalone
deployment.

Code

deploy/compose.yaml[20]

+  AGENT_SP_PERSISTENCE_PATH: /app/data/registrations

Evidence

The application treats the configured path as its durable JSON FileStore, but the Compose service
mounts only /kubeconfig; the only declared named volume belongs to NATS. The image creates
/app/data, which makes writes possible but does not preserve them across container replacement.

internal/config/config.go[54-58]
internal/provider/store/file.go[14-32]
deploy/compose.yaml[20-20]
deploy/compose.yaml[62-75]
Containerfile[21-23]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The configured registration store lives only in the environment-agent container writable layer and is lost when Compose recreates the container.

## Issue Context
Mount a named volume at `/app/data` (or its parent) while retaining a file path for `AGENT_SP_PERSISTENCE_PATH`.

## Fix Focus Areas
- deploy/compose.yaml[20-20]
- deploy/compose.yaml[62-64]
- deploy/compose.yaml[74-75]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

9. k8s-deploy applies stale hardcoded image before update ✓ Resolved 🐞 Bug ☼ Reliability

Description

k8s-deploy.sh runs kubectl apply -k (which creates/updates the Deployment using the manifest's
hardcoded quay.io/dcm-project/environment-agent:dev image with imagePullPolicy: IfNotPresent)
before calling kubectl set image with the actual built ${IMAGE}. When CONTAINER_IMAGE_NAME is
customized, this can cause Kubernetes to initially schedule a pod against the wrong/inaccessible
hardcoded image before the follow-up set image corrects it, risking a transient ImagePullBackOff.

Code

deploy/scripts/k8s-deploy.sh[R32-34]

+echo "==> Applying manifests"
+kubectl apply -k "${K8S_DIR}"
+kubectl -n dcm set image deployment/environment-agent environment-agent="${IMAGE}"

Evidence

deploy/k8s/environment-agent.yaml hardcodes image quay.io/dcm-project/environment-agent:dev with
IfNotPresent; k8s-deploy.sh applies the kustomization then only afterward runs kubectl set image
with the customized IMAGE, creating a window where the wrong image is referenced.

deploy/scripts/k8s-deploy.sh[32-34]
deploy/k8s/environment-agent.yaml[20-22]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
k8s-deploy.sh applies the kustomization (hardcoded :dev image) before setting the actual built image via kubectl set image, causing a transient window where the wrong image reference is used.

## Fix Focus Areas
- deploy/scripts/k8s-deploy.sh[32-34]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 17 rules

Review mode: 🧠 Deep: This is a broad deployment change spanning Compose, Kubernetes manifests/RBAC, Kind networking, shell automation, Make targets, and message publishing, creating many independent, easy-to-miss failure paths.

Tip of the day

💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[machacekondra]

All looks good to me. The only thing is that some files are redudant or similar with what we have in the control-plane repo.

Maybe it's time to think of separating this into e2e repo? Maybe we can have discussion about this.

[jenniferubah]

All looks good to me. The only thing is that some files are redudant or similar with what we have in the control-plane repo.

Maybe it's time to think of separating this into e2e repo? Maybe we can have discussion about this.

Sure, we can do the refactoring into separate repo in a follow up PR

[gciavarrini]

k8s-deploy.sh builds and loads ${IMAGE} from ENVIRONMENT_AGENT_VERSION, but kubectl apply -k still uses the hardcoded newTag: dev in kustomization. Anything other than dev loads the right image into Kind then schedules :dev.
Prefer kustomize edit set image or kubectl set image

[jenniferubah]

Done, d749d80

[gciavarrini]

In-cluster quick start uses plain kind create cluster.
On some hosts (Docker Desktop, rootless Podman) neither 127.0.0.1:30081 nor the node InternalIP is reachable, so make k8s-verify fails even when the pod is healthy. Maybe a small Kind config with extraPortMappings for 30081 and 30422? or documentation for port-forward as the fallback?

[jenniferubah]

added the extraPortMappings config and port-forwards as a fallback in the troubleshooting section - d749d80

[gciavarrini]

Default column says (in example) but .env.example keeps AGENT_EMBEDDED_SPS commented, which is fine as opt-in. Prefer empty (set in .env) or similar so the table matches the example file.

[jenniferubah]

updated, 5566f3e

[gciavarrini]

Compose defaults the agent image to :main, k8s-deploy to :dev. Each path builds and tags its own image, so switching from compose to in-cluster does not pick up the image you just built on the other path. Can we use one default tag for both workflows?

[jenniferubah]

Updated it to default to dev for both workflows. d749d80

[gciavarrini]

make kubeconfig-for-compose runs ./deploy/scripts/kubeconfig-for-compose.sh.

That needs the file to be executable so the target fails with permission denied on a fresh clone.
k8s-host-urls.sh avoids this by calling bash in the Makefile.

Prefer chmod +x here, or invoke via bash like the other scripts.

[jenniferubah]

Updated it to invoke bash - d749d80

[gciavarrini]

Comment says the tag should match k8s-deploy.sh, but the script never updates this file. Either wire the tag in the script or soften the comment so it does not imply automatic sync.

[jenniferubah]

Updated the comment

[gabriel-farache]

I had to change the permisions of the kubeconfig file to 644 instead of 600 as what is set in the kubeconfig-for-compose script to make it work; otherwise I had

{"time":"2026-09-07T18:59:35.38724248Z","level":"ERROR","msg":"failed to setup embedded SPs","error":"container embedded setup: creating container SP app: creating kubernetes client: building kubernetes config: error loading config file \"/kubeconfig\": open /kubeconfig: permission denied"}

Once this and the init script for NATS in the compose file is fixed, it works for me locally
I did not try to send message to the NATS topic of the agent to see

$ make deploy-verify 
bash deploy/scripts/verify.sh
==> Agent health (http://localhost:8081)
{"path":"health","status":"healthy"}

==> Registered providers (http://localhost:8081)
{"results":[{"create_time":"2026-09-07T19:08:51.938492438Z","endpoint":"embedded://container","id":"ce25d14c-6b07-4a97-bd5e-bb16698fcc25","last_check_time":"2026-09-07T19:08:51.981157985Z","name":"container","path":"providers/ce25d14c-6b07-4a97-bd5e-bb16698fcc25","schema_version":"v1alpha1","service_type":"container","status":"Unhealthy","type":"embedded","update_time":"2026-09-07T19:08:51.938492438Z"},{"create_time":"2026-09-07T19:08:51.976494151Z","endpoint":"embedded://vm","id":"c94ad4b3-94b8-4d43-a497-dda4edaeff4e","last_check_time":"2026-09-07T19:08:51.98115567Z","name":"vm","path":"providers/c94ad4b3-94b8-4d43-a497-dda4edaeff4e","schema_version":"v1alpha1","service_type":"vm","status":"Unhealthy","type":"embedded","update_time":"2026-09-07T19:08:51.976494151Z"}]}

[jenniferubah]

I had to change the permisions of the kubeconfig file to 644 instead of 600 as what is set in the kubeconfig-for-compose script to make it work; otherwise I had

{"time":"2026-09-07T18:59:35.38724248Z","level":"ERROR","msg":"failed to setup embedded SPs","error":"container embedded setup: creating container SP app: creating kubernetes client: building kubernetes config: error loading config file \"/kubeconfig\": open /kubeconfig: permission denied"}

Once this and the init script for NATS in the compose file is fixed, it works for me locally I did not try to send message to the NATS topic of the agent to see

$ make deploy-verify 
bash deploy/scripts/verify.sh
==> Agent health (http://localhost:8081)
{"path":"health","status":"healthy"}

==> Registered providers (http://localhost:8081)
{"results":[{"create_time":"2026-09-07T19:08:51.938492438Z","endpoint":"embedded://container","id":"ce25d14c-6b07-4a97-bd5e-bb16698fcc25","last_check_time":"2026-09-07T19:08:51.981157985Z","name":"container","path":"providers/ce25d14c-6b07-4a97-bd5e-bb16698fcc25","schema_version":"v1alpha1","service_type":"container","status":"Unhealthy","type":"embedded","update_time":"2026-09-07T19:08:51.938492438Z"},{"create_time":"2026-09-07T19:08:51.976494151Z","endpoint":"embedded://vm","id":"c94ad4b3-94b8-4d43-a497-dda4edaeff4e","last_check_time":"2026-09-07T19:08:51.98115567Z","name":"vm","path":"providers/c94ad4b3-94b8-4d43-a497-dda4edaeff4e","schema_version":"v1alpha1","service_type":"vm","status":"Unhealthy","type":"embedded","update_time":"2026-09-07T19:08:51.976494151Z"}]}

Thanks for testing this out 🙏🏾
Fixed the file mode in the utilites PR:
537e58b

[gabriel-farache]

can <node-ip> be an env var instead? if it is used in several places with env var we make sure it's the same everywhere (typo risk) and it will be easier for the user to just set the env and then copy/paste the command instead of having to update the command with the value

[jenniferubah]

I refactored this actually and added a command to do the export via the k8-host-urls script. Currently just one place this export with node-ip is used.

[gabriel-farache]

why removing this one?

[jenniferubah]

Great catch. This was mistakenly removed when removing the nats probe during testing.