From 901eb74b7f503a13aac16ce705c3e8b19006644c Mon Sep 17 00:00:00 2001 From: LukasWodka Date: Thu, 30 Jul 2026 14:06:02 +0200 Subject: [PATCH 1/5] feat(ingestor): resolve the spawned tag per environment (backend#1360) dev and staging edges now spawn ingestion Jobs from the internal channels published by the matching data-ingestors branch (:dev / :stg) instead of the 0.7 release float, so an ingestor change can be validated on a real edge without a production release. Before this, the ingestor image existed only as a byproduct of a prod release -- on 2026-07-30 testing one change cost a prod PyPI publish plus an FR-gate override. - images.ingestor.tag becomes an explicit override, EMPTY by default. - images.ingestor.channelTags carries the per-environment floats (dev/stg/prod), keyed on the resolved CLIENT_ENV. - New tracebloc.ingestorTag helper mirrors tracebloc.ingestorDigest's precedence: explicit tag > channel for CLIENT_ENV > literal 0.7 (so a release predating these keys still renders under --reuse-values). - prod deliberately stays a semver float, NOT a :prod channel -- no such tag is published, and prod normally runs prodDigest anyway. - The ingestor-multiarch CI guard previously hard-failed on an empty tag; it now validates the explicit override when set plus every channelTags entry, since an edge resolves exactly one of them. Chart defaults propagate through the fleet auto-upgrade (--reset-then-reuse-values), and the installer does not pin the tag, so existing dev/staging edges pick up their channel on the next upgrade. An operator who set images.ingestor.tag explicitly keeps it. Verified: 307/307 helm unittest across 27 suites (9 new cases covering each environment, the override, an unknown CLIENT_ENV, and a channelTags-less replay); helm lint --strict clean on all four platform values files; rendered INGESTOR_IMAGE_TAG confirmed as dev/stg/0.7/0.7 for dev/stg/prod/unset; schema still rejects 'latest' in channelTags and now accepts the empty override. --- .github/workflows/helm-ci.yaml | 24 +- client/templates/_helpers.tpl | 34 ++ client/templates/jobs-manager-deployment.yaml | 2 +- client/tests/ingestor_channel_tag_test.yaml | 99 ++++ client/tests/jobs_manager_test.yaml | 8 +- client/values.schema.json | 491 ++++++++++++++---- client/values.yaml | 26 +- 7 files changed, 576 insertions(+), 108 deletions(-) create mode 100644 client/tests/ingestor_channel_tag_test.yaml diff --git a/.github/workflows/helm-ci.yaml b/.github/workflows/helm-ci.yaml index 98b92ceb..2313270c 100644 --- a/.github/workflows/helm-ci.yaml +++ b/.github/workflows/helm-ci.yaml @@ -143,11 +143,27 @@ jobs: echo "OK — $label is multi-arch (amd64 + arm64)." } - # The floating tag is the default spawn target — always validate it. - if [ -z "$tag" ] || [ "$tag" = "null" ]; then - echo "::error::images.ingestor.tag is empty — the chart must define a floating tag to spawn by."; exit 1 + # Every tag this chart can spawn by must be multi-arch. Since + # backend#1360 that is `tag` (an explicit override, empty by default) + # PLUS each per-environment entry in `channelTags` — an edge resolves + # exactly one of them, so a single-arch value in any of them breaks + # ingestion on arm64 for whichever environment lands on it. + checked=0 + if [ -n "$tag" ] && [ "$tag" != "null" ]; then + assert_multiarch "${repo}:${tag}" "explicit tag override (images.ingestor.tag)" + checked=$((checked + 1)) + fi + for env_key in dev stg prod; do + channel=$(yq ".images.ingestor.channelTags.${env_key}" client/values.yaml) + if [ -z "$channel" ] || [ "$channel" = "null" ]; then + continue + fi + assert_multiarch "${repo}:${channel}" "channelTags.${env_key}" + checked=$((checked + 1)) + done + if [ "$checked" -eq 0 ]; then + echo "::error::no spawnable ingestor tag is defined — images.ingestor.tag is empty and channelTags has no entries, so an edge with no digest has nothing to spawn."; exit 1 fi - assert_multiarch "${repo}:${tag}" "floating tag" # The prod pin is a chart DEFAULT that reaches every prod edge through # the fleet auto-upgrade, so a single-arch value here would break diff --git a/client/templates/_helpers.tpl b/client/templates/_helpers.tpl index 67fda447..b57eeb4a 100644 --- a/client/templates/_helpers.tpl +++ b/client/templates/_helpers.tpl @@ -274,6 +274,40 @@ defeating the pin. Every read is nil-guarded for the same reason. Usage: {{ include "tracebloc.ingestorDigest" . }} */}} +{{/* + Effective floating tag for spawned ingestion Jobs (backend#1360). + + Precedence, mirroring tracebloc.ingestorDigest: + 1. `images.ingestor.tag` explicit override, any environment + 2. `images.ingestor.channelTags[CLIENT_ENV]` per-environment channel + 3. "0.7" last-resort literal, so a release that + predates these keys still renders under + `--reuse-values` + + Only consulted when no digest applies: jobs-manager builds `repo@digest` + when tracebloc.ingestorDigest is non-empty, and `repo:tag` otherwise + (client-runtime submit_ingestion_run._build_image_reference). + + dev/stg resolve to the UNSIGNED internal channels. Prod is a semver float, + not a `:prod` tag — none is published. +*/}} +{{- define "tracebloc.ingestorTag" -}} +{{- $ing := default dict .Values.images.ingestor -}} +{{- $explicit := $ing.tag | default "" -}} +{{- if $explicit -}} +{{- $explicit -}} +{{- else -}} +{{- $clientEnv := (default dict .Values.env).CLIENT_ENV | default "prod" -}} +{{- $channels := default dict $ing.channelTags -}} +{{- $channel := get $channels $clientEnv | default "" -}} +{{- if $channel -}} +{{- $channel -}} +{{- else -}} +{{- "0.7" -}} +{{- end -}} +{{- end -}} +{{- end }} + {{- define "tracebloc.ingestorDigest" -}} {{- $ing := default dict .Values.images.ingestor -}} {{- $explicit := $ing.digest | default "" -}} diff --git a/client/templates/jobs-manager-deployment.yaml b/client/templates/jobs-manager-deployment.yaml index d43674cb..fbdd12e2 100644 --- a/client/templates/jobs-manager-deployment.yaml +++ b/client/templates/jobs-manager-deployment.yaml @@ -134,7 +134,7 @@ spec: - name: INGESTOR_IMAGE_REPOSITORY value: {{ (default dict .Values.images.ingestor).repository | default "ghcr.io/tracebloc/ingestor" | quote }} - name: INGESTOR_IMAGE_TAG - value: {{ (default dict .Values.images.ingestor).tag | default "0.7" | quote }} + value: {{ include "tracebloc.ingestorTag" . | quote }} - name: INGESTOR_IMAGE_DIGEST value: {{ include "tracebloc.ingestorDigest" . | quote }} - name: REQUESTS_PROXY_URL diff --git a/client/tests/ingestor_channel_tag_test.yaml b/client/tests/ingestor_channel_tag_test.yaml new file mode 100644 index 00000000..03878689 --- /dev/null +++ b/client/tests/ingestor_channel_tag_test.yaml @@ -0,0 +1,99 @@ +suite: ingestor per-environment channel tag (backend#1360) +templates: + - templates/jobs-manager-deployment.yaml +release: + name: stg + namespace: tracebloc +set: + clientId: "test-id" + clientPassword: "test" +tests: + - it: dev edges spawn the :dev internal channel + set: + env.CLIENT_ENV: dev + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "dev" + + - it: staging edges spawn the :stg internal channel + set: + env.CLIENT_ENV: stg + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "stg" + + - it: prod stays on the semver float, never a :prod tag (none is published) + set: + env.CLIENT_ENV: prod + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "0.7" + + - it: defaults to the prod float when CLIENT_ENV is unset + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "0.7" + + - it: an explicit tag override wins over the channel for every environment + set: + env.CLIENT_ENV: dev + images.ingestor.tag: "0.8.0" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "0.8.0" + + - it: an unknown CLIENT_ENV falls back to the literal rather than rendering empty + set: + env.CLIENT_ENV: produktion + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "0.7" + + - it: renders for a release predating channelTags (--reuse-values replay) + set: + env.CLIENT_ENV: dev + images.ingestor.channelTags: null + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "0.7" + + - it: prod still pins by digest, so the tag is not the spawn target there + set: + env.CLIENT_ENV: prod + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_DIGEST + value: "sha256:9098b3c9b83951f825e262d6ff1e9c47c46c5b5cafbfc8fef1ed82ccbadebb24" + + - it: dev gets no digest, so it spawns by the floating channel tag + set: + env.CLIENT_ENV: dev + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_DIGEST + value: "" diff --git a/client/tests/jobs_manager_test.yaml b/client/tests/jobs_manager_test.yaml index e5390544..1a0009c1 100644 --- a/client/tests/jobs_manager_test.yaml +++ b/client/tests/jobs_manager_test.yaml @@ -123,7 +123,11 @@ tests: name: INGESTOR_IMAGE_DIGEST value: "sha256:9098b3c9b83951f825e262d6ff1e9c47c46c5b5cafbfc8fef1ed82ccbadebb24" - - it: keeps the ingestor floating on dev (CLIENT_ENV=dev => no digest) + - it: keeps the ingestor floating on dev (CLIENT_ENV=dev => no digest, :dev channel) + # Since backend#1360 the dev float is the per-environment INTERNAL + # channel (:dev) rather than the 0.7 release float — the point of the + # change: a dev edge tracks develop, so an ingestor change can be + # validated on a real edge without a production release. # dev/staging carry env.CLIENT_ENV in their user-supplied values, so the prod # pin does not apply: DIGEST empty => client-runtime spawns repo:tag with # imagePullPolicy=Always and the edge is never frozen on a stale build. @@ -140,7 +144,7 @@ tests: path: spec.template.spec.containers[0].env content: name: INGESTOR_IMAGE_TAG - value: "0.7" + value: "dev" - it: keeps the ingestor floating on staging (CLIENT_ENV=stg => no digest) set: diff --git a/client/values.schema.json b/client/values.schema.json index c6fb0314..9816bf76 100644 --- a/client/values.schema.json +++ b/client/values.schema.json @@ -2,7 +2,10 @@ "$schema": "https://json-schema.org/draft-07/schema#", "title": "Tracebloc Helm Chart Values", "type": "object", - "required": ["clientId", "clientPassword"], + "required": [ + "clientId", + "clientPassword" + ], "properties": { "env": { "type": "object", @@ -13,10 +16,18 @@ "default": "prod", "description": "Environment identifier and Docker image tag (dev, staging, prod). Defaults to 'prod' if not set or empty." }, - "HTTP_PROXY_HOST": { "type": "string" }, - "HTTP_PROXY_PORT": { "type": "string" }, - "HTTP_PROXY_USERNAME": { "type": "string" }, - "HTTP_PROXY_PASSWORD": { "type": "string" }, + "HTTP_PROXY_HOST": { + "type": "string" + }, + "HTTP_PROXY_PORT": { + "type": "string" + }, + "HTTP_PROXY_USERNAME": { + "type": "string" + }, + "HTTP_PROXY_PASSWORD": { + "type": "string" + }, "RESOURCE_REQUESTS": { "type": "string", "default": "cpu=2,memory=8Gi", @@ -39,7 +50,9 @@ "default": "nvidia.com/gpu=1", "description": "Optional GPU limits for spawned jobs (defaults to 'nvidia.com/gpu=1' if not set)" }, - "RUNTIME_CLASS_NAME": { "type": "string" }, + "RUNTIME_CLASS_NAME": { + "type": "string" + }, "SINGLE_NODE": { "type": "string", "description": "Single-node (fixed, non-elastic) cluster flag. Gates jobs-manager's GPU->CPU pending-job fallback (client-runtime#92). 'true' -> downgrade a stuck Pending GPU pod to CPU (correct on fixed single-host clusters where GPU presence is known at install time); 'false' -> leave it for the autoscaler/backend (correct for EKS/AKS/OpenShift). If unset, defaults to hostPath.enabled." @@ -51,7 +64,10 @@ }, "storageClass": { "type": "object", - "required": ["create", "name"], + "required": [ + "create", + "name" + ], "properties": { "create": { "type": "boolean", @@ -71,15 +87,25 @@ }, "volumeBindingMode": { "type": "string", - "enum": ["", "Immediate", "WaitForFirstConsumer"] + "enum": [ + "", + "Immediate", + "WaitForFirstConsumer" + ] }, "reclaimPolicy": { "type": "string", - "enum": ["", "Delete", "Retain"] + "enum": [ + "", + "Delete", + "Retain" + ] }, "mountOptions": { "type": "array", - "items": { "type": "string" } + "items": { + "type": "string" + } }, "parameters": { "type": "object", @@ -127,7 +153,11 @@ }, "pvcAccessMode": { "type": "string", - "enum": ["ReadWriteOnce", "ReadWriteMany", "ReadOnlyMany"], + "enum": [ + "ReadWriteOnce", + "ReadWriteMany", + "ReadOnlyMany" + ], "description": "Access mode for PVCs" }, "clusterScope": { @@ -146,7 +176,9 @@ "properties": { "namespace": { "type": "object", - "required": ["name"], + "required": [ + "name" + ], "properties": { "create": { "type": "boolean", @@ -191,21 +223,36 @@ "properties": { "warn": { "type": "string", - "enum": ["", "privileged", "baseline", "restricted"] + "enum": [ + "", + "privileged", + "baseline", + "restricted" + ] }, "warnVersion": { "type": "string" }, "audit": { "type": "string", - "enum": ["", "privileged", "baseline", "restricted"] + "enum": [ + "", + "privileged", + "baseline", + "restricted" + ] }, "auditVersion": { "type": "string" }, "enforce": { "type": "string", - "enum": ["", "privileged", "baseline", "restricted"] + "enum": [ + "", + "privileged", + "baseline", + "restricted" + ] }, "enforceVersion": { "type": "string" @@ -229,11 +276,11 @@ "allowExternalHttps": { "type": "boolean", "default": true, - "description": "When false, drop the 0.0.0.0/0:443 egress rule so training pods reach only DNS, MySQL, requests-proxy and the egress gateway (SECURITY §8.2 / client-runtime#102). Default true keeps existing behaviour; flip per-fleet after verifying the egress gateway works (G2)." + "description": "When false, drop the 0.0.0.0/0:443 egress rule so training pods reach only DNS, MySQL, requests-proxy and the egress gateway (SECURITY \u00a78.2 / client-runtime#102). Default true keeps existing behaviour; flip per-fleet after verifying the egress gateway works (G2)." }, "enforcementProbeHost": { "type": "string", - "description": "Host the `helm test` enforcement check curls directly (when allowExternalHttps=false) to verify the CNI blocks egress; non-enforcement fails the test (a test hook never affects install/upgrade) — client-runtime#104. Empty string disables it (e.g. air-gapped clusters)." + "description": "Host the `helm test` enforcement check curls directly (when allowExternalHttps=false) to verify the CNI blocks egress; non-enforcement fails the test (a test hook never affects install/upgrade) \u2014 client-runtime#104. Empty string disables it (e.g. air-gapped clusters)." }, "enforcementProbeTimeoutSeconds": { "type": "integer", @@ -255,7 +302,7 @@ }, "clusterCidrs": { "type": "array", - "description": "In-cluster pod+service CIDRs to block pod-to-pod egress on port 443. Override to match your cluster CIDRs. Must be non-empty when networkPolicy.training.enabled is true — an empty list renders `except: null` and Kubernetes treats that as no exceptions, silently allowing unrestricted in-cluster egress.", + "description": "In-cluster pod+service CIDRs to block pod-to-pod egress on port 443. Override to match your cluster CIDRs. Must be non-empty when networkPolicy.training.enabled is true \u2014 an empty list renders `except: null` and Kubernetes treats that as no exceptions, silently allowing unrestricted in-cluster egress.", "items": { "type": "string", "pattern": "^[0-9.]+/[0-9]+$" @@ -264,13 +311,19 @@ }, "if": { "properties": { - "enabled": { "const": true } + "enabled": { + "const": true + } }, - "required": ["enabled"] + "required": [ + "enabled" + ] }, "then": { "properties": { - "clusterCidrs": { "minItems": 1 } + "clusterCidrs": { + "minItems": 1 + } } } } @@ -317,7 +370,9 @@ }, "table_prefixes": { "type": "array", - "items": { "type": "string" }, + "items": { + "type": "string" + }, "description": "Table-name prefixes this SA may ingest into. [\"*\"] = any table." } } @@ -338,11 +393,11 @@ }, "sealCheck": { "type": "object", - "description": "Seal check — the chart's conformance suite (RFC-0003 §8.2 / backend#1184). Every check is a `helm test` hook Job labelled tracebloc.io/seal-check=true + tracebloc.io/seal-check-name= (the enumeration contract the tracebloc CLI consumes, cli#393). See docs/SEAL-CHECK.md.", + "description": "Seal check \u2014 the chart's conformance suite (RFC-0003 \u00a78.2 / backend#1184). Every check is a `helm test` hook Job labelled tracebloc.io/seal-check=true + tracebloc.io/seal-check-name= (the enumeration contract the tracebloc CLI consumes, cli#393). See docs/SEAL-CHECK.md.", "properties": { "storageAssertions": { "type": "object", - "description": "In-cluster storage assertions: release PVCs Bound on the expected StorageClass; in dynamic-PVC mode (hostPath.enabled=false) no release PVC backed by a hostPath PV on an unmanaged host tree (leftover chart hostPath PVs capture claims via claimRef — RFC-0003 D3/D4).", + "description": "In-cluster storage assertions: release PVCs Bound on the expected StorageClass; in dynamic-PVC mode (hostPath.enabled=false) no release PVC backed by a hostPath PV on an unmanaged host tree (leftover chart hostPath PVs capture claims via claimRef \u2014 RFC-0003 D3/D4).", "properties": { "enabled": { "type": "boolean", @@ -357,16 +412,35 @@ }, "nodeLocalPathPrefixes": { "type": "array", - "items": { "type": "string" }, - "description": "hostPath prefixes accepted as provisioner-managed node-local storage in dynamic mode (k3s local-path: /var/lib/rancher/…; upstream local-path-provisioner: /opt/local-path-provisioner/). Entries match whole path segments (trailing slash optional) — a prefix admits itself and paths under it, never sibling paths. Any other hostPath backing fails the check." + "items": { + "type": "string" + }, + "description": "hostPath prefixes accepted as provisioner-managed node-local storage in dynamic mode (k3s local-path: /var/lib/rancher/\u2026; upstream local-path-provisioner: /opt/local-path-provisioner/). Entries match whole path segments (trailing slash optional) \u2014 a prefix admits itself and paths under it, never sibling paths. Any other hostPath backing fails the check." }, "image": { "type": "object", "description": "kubectl-capable image for the assertion pod (same image the image-refresh CronJob uses).", "properties": { - "repository": { "type": "string", "default": "alpine/k8s" }, - "tag": { "type": "string", "not": { "const": "latest" }, "default": "1.30.5" }, - "pullPolicy": { "type": "string", "enum": ["Always", "IfNotPresent", "Never"], "default": "IfNotPresent" } + "repository": { + "type": "string", + "default": "alpine/k8s" + }, + "tag": { + "type": "string", + "not": { + "const": "latest" + }, + "default": "1.30.5" + }, + "pullPolicy": { + "type": "string", + "enum": [ + "Always", + "IfNotPresent", + "Never" + ], + "default": "IfNotPresent" + } } } } @@ -375,25 +449,56 @@ }, "egressProxy": { "type": "object", - "description": "In-cluster squid egress gateway (SECURITY §8.2 / client-runtime#102). Forward proxy that permits HTTPS CONNECT only to the allowlist, so a locked-down training pod can reach the backend + App Insights and nothing else.", + "description": "In-cluster squid egress gateway (SECURITY \u00a78.2 / client-runtime#102). Forward proxy that permits HTTPS CONNECT only to the allowlist, so a locked-down training pod can reach the backend + App Insights and nothing else.", "properties": { - "enabled": { "type": "boolean", "default": true }, - "routeWorkloads": { "type": "boolean", "default": false, "description": "Route training-pod outbound HTTPS through the gateway (jobs-manager injects HTTPS_PROXY). Default false — enable per-fleet, verify a run, then drop the direct egress rule (networkPolicy.training.allowExternalHttps=false)." }, - "port": { "type": "integer", "minimum": 1, "maximum": 65535, "default": 3128 }, - "runAsUser": { "type": "integer", "minimum": 1 }, + "enabled": { + "type": "boolean", + "default": true + }, + "routeWorkloads": { + "type": "boolean", + "default": false, + "description": "Route training-pod outbound HTTPS through the gateway (jobs-manager injects HTTPS_PROXY). Default false \u2014 enable per-fleet, verify a run, then drop the direct egress rule (networkPolicy.training.allowExternalHttps=false)." + }, + "port": { + "type": "integer", + "minimum": 1, + "maximum": 65535, + "default": 3128 + }, + "runAsUser": { + "type": "integer", + "minimum": 1 + }, "image": { "type": "object", "properties": { - "registry": { "type": "string" }, - "repository": { "type": "string", "minLength": 1 }, - "tag": { "type": "string", "not": { "const": "latest" } }, - "digest": { "type": "string", "pattern": "^(sha256:[a-f0-9]{64})?$" } + "registry": { + "type": "string" + }, + "repository": { + "type": "string", + "minLength": 1 + }, + "tag": { + "type": "string", + "not": { + "const": "latest" + } + }, + "digest": { + "type": "string", + "pattern": "^(sha256:[a-f0-9]{64})?$" + } } }, "allowlist": { "type": "array", "description": "FQDNs the gateway permits HTTPS CONNECT to (squid dstdomain syntax: leading dot = subdomain match, bare host = exact).", - "items": { "type": "string", "minLength": 1 } + "items": { + "type": "string", + "minLength": 1 + } }, "resources": { "type": "object", @@ -401,15 +506,27 @@ "requests": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } }, "limits": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } } } @@ -459,7 +576,7 @@ }, "ingestor": { "type": "object", - "description": "Image used for spawned ingestion Jobs. jobs-manager resolves one effective digest from these keys (see the tracebloc.ingestorDigest helper) and builds the reference from it (client-runtime submit_ingestion_run): digest set -> repo@digest + IfNotPresent, digest empty -> repo:tag + Always. Prod pins `prodDigest` (a chart default, so a republished pin propagates through the fleet auto-upgrade — backend#1245); dev/staging float on `tag`, resolving the current published digest at every spawn.", + "description": "Image used for spawned ingestion Jobs. jobs-manager resolves one effective digest from these keys (see the tracebloc.ingestorDigest helper) and builds the reference from it (client-runtime submit_ingestion_run): digest set -> repo@digest + IfNotPresent, digest empty -> repo:tag + Always. Prod pins `prodDigest` (a chart default, so a republished pin propagates through the fleet auto-upgrade \u2014 backend#1245); dev/staging float on `tag`, resolving the current published digest at every spawn.", "properties": { "repository": { "type": "string", @@ -468,9 +585,10 @@ }, "tag": { "type": "string", - "minLength": 1, - "not": { "const": "latest" }, - "description": "Floating ghcr.io/tracebloc/ingestor tag spawned when no digest applies (imagePullPolicy=Always). `latest` is rejected — spawn behaviour must not drift. Surfaces as INGESTOR_IMAGE_TAG." + "not": { + "const": "latest" + }, + "description": "Explicit floating-tag override for the spawned ingestor. EMPTY by default: the effective tag then comes from channelTags[CLIENT_ENV]. `latest` is rejected \u2014 spawn behaviour must not drift. To pin an exact image prefer `digest`, which wins over any tag. Surfaces as INGESTOR_IMAGE_TAG." }, "prodDigest": { "type": "string", @@ -485,6 +603,36 @@ "type": "string", "pattern": "^(sha256:[a-f0-9]{64})?$", "description": "Optional explicit per-edge pin (sha256:<64 hex>). Empty (default) = let prodPin/CLIENT_ENV decide. When set it wins over `prodDigest` and over `prodPin: false` in ANY environment, and every ingestion Job runs this exact image with imagePullPolicy=IfNotPresent (reproducibility). Must be a multi-arch index (#186). Surfaces as INGESTOR_IMAGE_DIGEST." + }, + "channelTags": { + "type": "object", + "description": "Per-environment floating tags, keyed on the resolved CLIENT_ENV, used when `tag` is empty. dev/stg track the unsigned internal channels published from the matching data-ingestors branch; prod stays a semver float (no :prod tag is published). backend#1360.", + "properties": { + "dev": { + "type": "string", + "minLength": 1, + "not": { + "const": "latest" + }, + "description": "Tag spawned when CLIENT_ENV resolves to dev." + }, + "stg": { + "type": "string", + "minLength": 1, + "not": { + "const": "latest" + }, + "description": "Tag spawned when CLIENT_ENV resolves to stg." + }, + "prod": { + "type": "string", + "minLength": 1, + "not": { + "const": "latest" + }, + "description": "Tag spawned when CLIENT_ENV resolves to prod." + } + } } } }, @@ -493,7 +641,9 @@ "properties": { "tag": { "type": "string", - "not": { "const": "latest" }, + "not": { + "const": "latest" + }, "description": "Image tag for mysql-client. Empty falls back to env.CLIENT_ENV. Must not be \"latest\"." }, "digest": { @@ -507,7 +657,9 @@ "properties": { "tag": { "type": "string", - "not": { "const": "latest" } + "not": { + "const": "latest" + } }, "digest": { "type": "string", @@ -527,16 +679,28 @@ "requests": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } }, "limits": { "type": "object", - "description": "CPU limit deliberately omitted in defaults — throttling causes InnoDB lock-wait timeouts.", + "description": "CPU limit deliberately omitted in defaults \u2014 throttling causes InnoDB lock-wait timeouts.", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } } } @@ -547,15 +711,27 @@ "requests": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } }, "limits": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } } } @@ -566,15 +742,27 @@ "requests": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } }, "limits": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } } } @@ -585,15 +773,27 @@ "requests": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } }, "limits": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } } } @@ -602,21 +802,42 @@ }, "priorityClass": { "type": "object", - "description": "Cluster-scoped PriorityClass for tracebloc data-plane workloads (mysql). An empty `name` disables the priorityClassName reference on the mysql pod entirely (mysql falls back to default priority 0 — only set this if you explicitly want no OOM protection from the scheduler).", + "description": "Cluster-scoped PriorityClass for tracebloc data-plane workloads (mysql). An empty `name` disables the priorityClassName reference on the mysql pod entirely (mysql falls back to default priority 0 \u2014 only set this if you explicitly want no OOM protection from the scheduler).", "properties": { - "create": { "type": "boolean", "default": true }, - "name": { "type": "string", "default": "tracebloc-data-plane" }, - "value": { "type": "integer", "minimum": 0, "maximum": 1000000000, "default": 1000000 } + "create": { + "type": "boolean", + "default": true + }, + "name": { + "type": "string", + "default": "tracebloc-data-plane" + }, + "value": { + "type": "integer", + "minimum": 0, + "maximum": 1000000000, + "default": 1000000 + } }, "allOf": [ { - "$comment": "If we are templating the PriorityClass resource, name must not be empty. `required: [create]` is needed because draft-07 `properties` only validates keys that are present — without it, omitting `create` passes the `if` vacuously and unconditionally enforces `then`. Same pattern as dockerRegistry below.", + "$comment": "If we are templating the PriorityClass resource, name must not be empty. `required: [create]` is needed because draft-07 `properties` only validates keys that are present \u2014 without it, omitting `create` passes the `if` vacuously and unconditionally enforces `then`. Same pattern as dockerRegistry below.", "if": { - "properties": { "create": { "const": true } }, - "required": ["create"] + "properties": { + "create": { + "const": true + } + }, + "required": [ + "create" + ] }, "then": { - "properties": { "name": { "minLength": 1 } } + "properties": { + "name": { + "minLength": 1 + } + } } } ] @@ -628,13 +849,19 @@ "mysql": { "type": "object", "properties": { - "create": { "type": "boolean", "default": true } + "create": { + "type": "boolean", + "default": true + } } }, "jobsManager": { "type": "object", "properties": { - "create": { "type": "boolean", "default": true } + "create": { + "type": "boolean", + "default": true + } } } } @@ -642,13 +869,17 @@ "clientId": { "type": "string", "minLength": 1, - "not": { "pattern": "^<.*>$" }, + "not": { + "pattern": "^<.*>$" + }, "description": "Client ID for authentication. Must be a real value, not a placeholder like ." }, "clientPassword": { "type": "string", "minLength": 1, - "not": { "pattern": "^<.*>$" }, + "not": { + "pattern": "^<.*>$" + }, "description": "Client authentication password. Must be a real value, not a placeholder like ." }, "autoUpgrade": { @@ -704,18 +935,30 @@ }, "image": { "type": "object", - "required": ["repository", "tag"], + "required": [ + "repository", + "tag" + ], "properties": { - "repository": { "type": "string", "minLength": 1 }, + "repository": { + "type": "string", + "minLength": 1 + }, "tag": { "type": "string", "minLength": 1, - "not": { "const": "latest" }, - "description": "Pin the helm version. `latest` is rejected — auto-upgrade behaviour must not drift." + "not": { + "const": "latest" + }, + "description": "Pin the helm version. `latest` is rejected \u2014 auto-upgrade behaviour must not drift." }, "pullPolicy": { "type": "string", - "enum": ["Always", "IfNotPresent", "Never"] + "enum": [ + "Always", + "IfNotPresent", + "Never" + ] } } }, @@ -725,15 +968,27 @@ "requests": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } }, "limits": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } } } @@ -778,18 +1033,30 @@ }, "image": { "type": "object", - "required": ["repository", "tag"], + "required": [ + "repository", + "tag" + ], "properties": { - "repository": { "type": "string", "minLength": 1 }, + "repository": { + "type": "string", + "minLength": 1 + }, "tag": { "type": "string", "minLength": 1, - "not": { "const": "latest" }, - "description": "Pin the alpine/k8s version. `latest` is rejected — image-refresh behaviour must not drift." + "not": { + "const": "latest" + }, + "description": "Pin the alpine/k8s version. `latest` is rejected \u2014 image-refresh behaviour must not drift." }, "pullPolicy": { "type": "string", - "enum": ["Always", "IfNotPresent", "Never"] + "enum": [ + "Always", + "IfNotPresent", + "Never" + ] } } }, @@ -799,15 +1066,27 @@ "requests": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } }, "limits": { "type": "object", "properties": { - "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, - "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } + "cpu": { + "type": "string", + "pattern": "^[0-9]+m?$" + }, + "memory": { + "type": "string", + "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" + } } } } @@ -815,7 +1094,10 @@ } }, "dockerRegistry": { - "type": ["object", "null"], + "type": [ + "object", + "null" + ], "description": "Optional. Omit entirely or set null for public images (no secret or imagePullSecrets). Only create when set and create is true.", "properties": { "create": { @@ -841,12 +1123,21 @@ { "if": { "properties": { - "create": { "const": true } + "create": { + "const": true + } }, - "required": ["create"] + "required": [ + "create" + ] }, "then": { - "required": ["server", "username", "password", "email"] + "required": [ + "server", + "username", + "password", + "email" + ] } } ] diff --git a/client/values.yaml b/client/values.yaml index 631a11fa..26e47c3c 100644 --- a/client/values.yaml +++ b/client/values.yaml @@ -389,7 +389,31 @@ images: # newer categories re-opens the submit-vs-run drift bug (categories # accepted at submit that the spawned image can't process). See # client-runtime#162 discussion. - tag: "0.7" + # Empty by DEFAULT: the effective tag now comes from `channelTags` below, + # keyed on the resolved CLIENT_ENV. Set this to force one tag on an edge + # regardless of environment (to pin an exact immutable image, prefer + # `digest` — it wins over any tag). backend#1360. + tag: "" + + # -- PER-ENVIRONMENT CHANNELS (backend#1360) + # + # dev and staging edges track the internal channels published from the + # matching data-ingestors branch (`publish-images.yml`: develop -> :dev, + # staging -> :stg), so an ingestor change can be validated on a real edge + # WITHOUT a production release. Before this, the ingestor image existed + # only as a byproduct of a prod release, so testing one change cost a prod + # PyPI publish plus an FR-gate override (2026-07-30). + # + # `prod` deliberately stays a semver float, NOT a `:prod` channel — no + # such tag is published; prod normally runs `prodDigest` anyway and this is + # only the fallback when pinning is disabled. Keep the prod entry tracking + # the current release line for the same submit-vs-run reasons noted above. + # + # These are unsigned internal images. Do not point a prod edge at them. + channelTags: + dev: "dev" + stg: "stg" + prod: "0.7" # -- PROD REPRODUCIBILITY PIN — the digest prod edges run. # From 6c207d6ddf122dbf0dc962251e1915931ce8f969 Mon Sep 17 00:00:00 2001 From: LukasWodka Date: Thu, 30 Jul 2026 16:55:13 +0200 Subject: [PATCH 2/5] fix(review): normalize CLIENT_ENV aliases before the channel lookup Asad's first note landed on a real defect, not just a stale description. The schema documents CLIENT_ENV as (dev, staging, prod) while the channel keys are dev|stg|prod, so CLIENT_ENV=staging -- the documented value -- missed channelTags entirely and fell back to the prod float. Meanwhile client-runtime normalizes staging->stg at runtime, so that edge would have talked to the stg backend while spawning the 0.7 release ingestor: exactly the split-brain client-runtime#227 was filed for, reintroduced one layer up. tracebloc.ingestorTag now normalizes development/staging/production before the lookup, mirroring proxy_config.ENV_ALIASES, and the schema description states which values are canonical and that it is load-bearing for channel selection. Also addresses the second note: values.schema.json is edited surgically in the file's own style instead of being reformatted by a json round-trip (576-line diff -> 29). Verified: 310/310 helm unittest (3 new alias cases); lint --strict clean on all four platform files; rendered tag is dev/dev/stg/stg/0.7/0.7 for dev/development/stg/staging/prod/production, and an unknown value still falls back to the float rather than rendering empty. --- client/templates/_helpers.tpl | 14 + client/tests/ingestor_channel_tag_test.yaml | 34 ++ client/values.schema.json | 506 +++++--------------- 3 files changed, 167 insertions(+), 387 deletions(-) diff --git a/client/templates/_helpers.tpl b/client/templates/_helpers.tpl index b57eeb4a..c7170664 100644 --- a/client/templates/_helpers.tpl +++ b/client/templates/_helpers.tpl @@ -298,6 +298,20 @@ Usage: {{ include "tracebloc.ingestorDigest" . }} {{- $explicit -}} {{- else -}} {{- $clientEnv := (default dict .Values.env).CLIENT_ENV | default "prod" -}} +{{/* + Normalize the documented aliases before the lookup. The chart docs and the + SDK say dev|staging|prod while these keys are dev|stg|prod, and + client-runtime normalizes the same three at runtime + (proxy_config.ENV_ALIASES). Without this, CLIENT_ENV=staging -- the value + the schema documents -- missed channelTags entirely and fell back to the + prod float: the service would talk to the stg backend while spawning the + release ingestor, which is precisely the split-brain client-runtime#227 + was filed for. Review catch on client#494. +*/}} +{{- $aliases := dict "development" "dev" "staging" "stg" "production" "prod" -}} +{{- if hasKey $aliases $clientEnv -}} +{{- $clientEnv = get $aliases $clientEnv -}} +{{- end -}} {{- $channels := default dict $ing.channelTags -}} {{- $channel := get $channels $clientEnv | default "" -}} {{- if $channel -}} diff --git a/client/tests/ingestor_channel_tag_test.yaml b/client/tests/ingestor_channel_tag_test.yaml index 03878689..baf7d667 100644 --- a/client/tests/ingestor_channel_tag_test.yaml +++ b/client/tests/ingestor_channel_tag_test.yaml @@ -97,3 +97,37 @@ tests: content: name: INGESTOR_IMAGE_DIGEST value: "" + + - it: the documented alias "staging" resolves to the stg channel, not the prod float + # Review catch (#494): the schema documents dev|staging|prod while these + # keys are dev|stg|prod, so CLIENT_ENV=staging previously missed + # channelTags and silently fell back to 0.7 while client-runtime + # normalized it to stg at runtime — the split-brain of client-runtime#227. + set: + env.CLIENT_ENV: staging + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "stg" + + - it: the documented alias "development" resolves to the dev channel + set: + env.CLIENT_ENV: development + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "dev" + + - it: the documented alias "production" resolves to the prod float + set: + env.CLIENT_ENV: production + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_TAG + value: "0.7" diff --git a/client/values.schema.json b/client/values.schema.json index 9816bf76..c734975a 100644 --- a/client/values.schema.json +++ b/client/values.schema.json @@ -2,10 +2,7 @@ "$schema": "https://json-schema.org/draft-07/schema#", "title": "Tracebloc Helm Chart Values", "type": "object", - "required": [ - "clientId", - "clientPassword" - ], + "required": ["clientId", "clientPassword"], "properties": { "env": { "type": "object", @@ -14,20 +11,12 @@ "CLIENT_ENV": { "type": "string", "default": "prod", - "description": "Environment identifier and Docker image tag (dev, staging, prod). Defaults to 'prod' if not set or empty." - }, - "HTTP_PROXY_HOST": { - "type": "string" - }, - "HTTP_PROXY_PORT": { - "type": "string" - }, - "HTTP_PROXY_USERNAME": { - "type": "string" - }, - "HTTP_PROXY_PASSWORD": { - "type": "string" + "description": "Environment identifier and Docker image tag. Canonical values are `dev`, `stg`, `prod`; the documented aliases `development`/`staging`/`production` are accepted and normalized (matching client-runtime proxy_config.ENV_ALIASES). Load-bearing: it selects images.ingestor.channelTags and the prod digest pin. Defaults to 'prod' if not set or empty." }, + "HTTP_PROXY_HOST": { "type": "string" }, + "HTTP_PROXY_PORT": { "type": "string" }, + "HTTP_PROXY_USERNAME": { "type": "string" }, + "HTTP_PROXY_PASSWORD": { "type": "string" }, "RESOURCE_REQUESTS": { "type": "string", "default": "cpu=2,memory=8Gi", @@ -50,9 +39,7 @@ "default": "nvidia.com/gpu=1", "description": "Optional GPU limits for spawned jobs (defaults to 'nvidia.com/gpu=1' if not set)" }, - "RUNTIME_CLASS_NAME": { - "type": "string" - }, + "RUNTIME_CLASS_NAME": { "type": "string" }, "SINGLE_NODE": { "type": "string", "description": "Single-node (fixed, non-elastic) cluster flag. Gates jobs-manager's GPU->CPU pending-job fallback (client-runtime#92). 'true' -> downgrade a stuck Pending GPU pod to CPU (correct on fixed single-host clusters where GPU presence is known at install time); 'false' -> leave it for the autoscaler/backend (correct for EKS/AKS/OpenShift). If unset, defaults to hostPath.enabled." @@ -64,10 +51,7 @@ }, "storageClass": { "type": "object", - "required": [ - "create", - "name" - ], + "required": ["create", "name"], "properties": { "create": { "type": "boolean", @@ -87,25 +71,15 @@ }, "volumeBindingMode": { "type": "string", - "enum": [ - "", - "Immediate", - "WaitForFirstConsumer" - ] + "enum": ["", "Immediate", "WaitForFirstConsumer"] }, "reclaimPolicy": { "type": "string", - "enum": [ - "", - "Delete", - "Retain" - ] + "enum": ["", "Delete", "Retain"] }, "mountOptions": { "type": "array", - "items": { - "type": "string" - } + "items": { "type": "string" } }, "parameters": { "type": "object", @@ -153,11 +127,7 @@ }, "pvcAccessMode": { "type": "string", - "enum": [ - "ReadWriteOnce", - "ReadWriteMany", - "ReadOnlyMany" - ], + "enum": ["ReadWriteOnce", "ReadWriteMany", "ReadOnlyMany"], "description": "Access mode for PVCs" }, "clusterScope": { @@ -176,9 +146,7 @@ "properties": { "namespace": { "type": "object", - "required": [ - "name" - ], + "required": ["name"], "properties": { "create": { "type": "boolean", @@ -223,36 +191,21 @@ "properties": { "warn": { "type": "string", - "enum": [ - "", - "privileged", - "baseline", - "restricted" - ] + "enum": ["", "privileged", "baseline", "restricted"] }, "warnVersion": { "type": "string" }, "audit": { "type": "string", - "enum": [ - "", - "privileged", - "baseline", - "restricted" - ] + "enum": ["", "privileged", "baseline", "restricted"] }, "auditVersion": { "type": "string" }, "enforce": { "type": "string", - "enum": [ - "", - "privileged", - "baseline", - "restricted" - ] + "enum": ["", "privileged", "baseline", "restricted"] }, "enforceVersion": { "type": "string" @@ -276,11 +229,11 @@ "allowExternalHttps": { "type": "boolean", "default": true, - "description": "When false, drop the 0.0.0.0/0:443 egress rule so training pods reach only DNS, MySQL, requests-proxy and the egress gateway (SECURITY \u00a78.2 / client-runtime#102). Default true keeps existing behaviour; flip per-fleet after verifying the egress gateway works (G2)." + "description": "When false, drop the 0.0.0.0/0:443 egress rule so training pods reach only DNS, MySQL, requests-proxy and the egress gateway (SECURITY §8.2 / client-runtime#102). Default true keeps existing behaviour; flip per-fleet after verifying the egress gateway works (G2)." }, "enforcementProbeHost": { "type": "string", - "description": "Host the `helm test` enforcement check curls directly (when allowExternalHttps=false) to verify the CNI blocks egress; non-enforcement fails the test (a test hook never affects install/upgrade) \u2014 client-runtime#104. Empty string disables it (e.g. air-gapped clusters)." + "description": "Host the `helm test` enforcement check curls directly (when allowExternalHttps=false) to verify the CNI blocks egress; non-enforcement fails the test (a test hook never affects install/upgrade) — client-runtime#104. Empty string disables it (e.g. air-gapped clusters)." }, "enforcementProbeTimeoutSeconds": { "type": "integer", @@ -302,7 +255,7 @@ }, "clusterCidrs": { "type": "array", - "description": "In-cluster pod+service CIDRs to block pod-to-pod egress on port 443. Override to match your cluster CIDRs. Must be non-empty when networkPolicy.training.enabled is true \u2014 an empty list renders `except: null` and Kubernetes treats that as no exceptions, silently allowing unrestricted in-cluster egress.", + "description": "In-cluster pod+service CIDRs to block pod-to-pod egress on port 443. Override to match your cluster CIDRs. Must be non-empty when networkPolicy.training.enabled is true — an empty list renders `except: null` and Kubernetes treats that as no exceptions, silently allowing unrestricted in-cluster egress.", "items": { "type": "string", "pattern": "^[0-9.]+/[0-9]+$" @@ -311,19 +264,13 @@ }, "if": { "properties": { - "enabled": { - "const": true - } + "enabled": { "const": true } }, - "required": [ - "enabled" - ] + "required": ["enabled"] }, "then": { "properties": { - "clusterCidrs": { - "minItems": 1 - } + "clusterCidrs": { "minItems": 1 } } } } @@ -370,9 +317,7 @@ }, "table_prefixes": { "type": "array", - "items": { - "type": "string" - }, + "items": { "type": "string" }, "description": "Table-name prefixes this SA may ingest into. [\"*\"] = any table." } } @@ -393,11 +338,11 @@ }, "sealCheck": { "type": "object", - "description": "Seal check \u2014 the chart's conformance suite (RFC-0003 \u00a78.2 / backend#1184). Every check is a `helm test` hook Job labelled tracebloc.io/seal-check=true + tracebloc.io/seal-check-name= (the enumeration contract the tracebloc CLI consumes, cli#393). See docs/SEAL-CHECK.md.", + "description": "Seal check — the chart's conformance suite (RFC-0003 §8.2 / backend#1184). Every check is a `helm test` hook Job labelled tracebloc.io/seal-check=true + tracebloc.io/seal-check-name= (the enumeration contract the tracebloc CLI consumes, cli#393). See docs/SEAL-CHECK.md.", "properties": { "storageAssertions": { "type": "object", - "description": "In-cluster storage assertions: release PVCs Bound on the expected StorageClass; in dynamic-PVC mode (hostPath.enabled=false) no release PVC backed by a hostPath PV on an unmanaged host tree (leftover chart hostPath PVs capture claims via claimRef \u2014 RFC-0003 D3/D4).", + "description": "In-cluster storage assertions: release PVCs Bound on the expected StorageClass; in dynamic-PVC mode (hostPath.enabled=false) no release PVC backed by a hostPath PV on an unmanaged host tree (leftover chart hostPath PVs capture claims via claimRef — RFC-0003 D3/D4).", "properties": { "enabled": { "type": "boolean", @@ -412,35 +357,16 @@ }, "nodeLocalPathPrefixes": { "type": "array", - "items": { - "type": "string" - }, - "description": "hostPath prefixes accepted as provisioner-managed node-local storage in dynamic mode (k3s local-path: /var/lib/rancher/\u2026; upstream local-path-provisioner: /opt/local-path-provisioner/). Entries match whole path segments (trailing slash optional) \u2014 a prefix admits itself and paths under it, never sibling paths. Any other hostPath backing fails the check." + "items": { "type": "string" }, + "description": "hostPath prefixes accepted as provisioner-managed node-local storage in dynamic mode (k3s local-path: /var/lib/rancher/…; upstream local-path-provisioner: /opt/local-path-provisioner/). Entries match whole path segments (trailing slash optional) — a prefix admits itself and paths under it, never sibling paths. Any other hostPath backing fails the check." }, "image": { "type": "object", "description": "kubectl-capable image for the assertion pod (same image the image-refresh CronJob uses).", "properties": { - "repository": { - "type": "string", - "default": "alpine/k8s" - }, - "tag": { - "type": "string", - "not": { - "const": "latest" - }, - "default": "1.30.5" - }, - "pullPolicy": { - "type": "string", - "enum": [ - "Always", - "IfNotPresent", - "Never" - ], - "default": "IfNotPresent" - } + "repository": { "type": "string", "default": "alpine/k8s" }, + "tag": { "type": "string", "not": { "const": "latest" }, "default": "1.30.5" }, + "pullPolicy": { "type": "string", "enum": ["Always", "IfNotPresent", "Never"], "default": "IfNotPresent" } } } } @@ -449,56 +375,25 @@ }, "egressProxy": { "type": "object", - "description": "In-cluster squid egress gateway (SECURITY \u00a78.2 / client-runtime#102). Forward proxy that permits HTTPS CONNECT only to the allowlist, so a locked-down training pod can reach the backend + App Insights and nothing else.", + "description": "In-cluster squid egress gateway (SECURITY §8.2 / client-runtime#102). Forward proxy that permits HTTPS CONNECT only to the allowlist, so a locked-down training pod can reach the backend + App Insights and nothing else.", "properties": { - "enabled": { - "type": "boolean", - "default": true - }, - "routeWorkloads": { - "type": "boolean", - "default": false, - "description": "Route training-pod outbound HTTPS through the gateway (jobs-manager injects HTTPS_PROXY). Default false \u2014 enable per-fleet, verify a run, then drop the direct egress rule (networkPolicy.training.allowExternalHttps=false)." - }, - "port": { - "type": "integer", - "minimum": 1, - "maximum": 65535, - "default": 3128 - }, - "runAsUser": { - "type": "integer", - "minimum": 1 - }, + "enabled": { "type": "boolean", "default": true }, + "routeWorkloads": { "type": "boolean", "default": false, "description": "Route training-pod outbound HTTPS through the gateway (jobs-manager injects HTTPS_PROXY). Default false — enable per-fleet, verify a run, then drop the direct egress rule (networkPolicy.training.allowExternalHttps=false)." }, + "port": { "type": "integer", "minimum": 1, "maximum": 65535, "default": 3128 }, + "runAsUser": { "type": "integer", "minimum": 1 }, "image": { "type": "object", "properties": { - "registry": { - "type": "string" - }, - "repository": { - "type": "string", - "minLength": 1 - }, - "tag": { - "type": "string", - "not": { - "const": "latest" - } - }, - "digest": { - "type": "string", - "pattern": "^(sha256:[a-f0-9]{64})?$" - } + "registry": { "type": "string" }, + "repository": { "type": "string", "minLength": 1 }, + "tag": { "type": "string", "not": { "const": "latest" } }, + "digest": { "type": "string", "pattern": "^(sha256:[a-f0-9]{64})?$" } } }, "allowlist": { "type": "array", "description": "FQDNs the gateway permits HTTPS CONNECT to (squid dstdomain syntax: leading dot = subdomain match, bare host = exact).", - "items": { - "type": "string", - "minLength": 1 - } + "items": { "type": "string", "minLength": 1 } }, "resources": { "type": "object", @@ -506,27 +401,15 @@ "requests": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } }, "limits": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } } } @@ -576,7 +459,7 @@ }, "ingestor": { "type": "object", - "description": "Image used for spawned ingestion Jobs. jobs-manager resolves one effective digest from these keys (see the tracebloc.ingestorDigest helper) and builds the reference from it (client-runtime submit_ingestion_run): digest set -> repo@digest + IfNotPresent, digest empty -> repo:tag + Always. Prod pins `prodDigest` (a chart default, so a republished pin propagates through the fleet auto-upgrade \u2014 backend#1245); dev/staging float on `tag`, resolving the current published digest at every spawn.", + "description": "Image used for spawned ingestion Jobs. jobs-manager resolves one effective digest from these keys (see the tracebloc.ingestorDigest helper) and builds the reference from it (client-runtime submit_ingestion_run): digest set -> repo@digest + IfNotPresent, digest empty -> repo:tag + Always. Prod pins `prodDigest` (a chart default, so a republished pin propagates through the fleet auto-upgrade — backend#1245); dev/staging float on `tag`, resolving the current published digest at every spawn.", "properties": { "repository": { "type": "string", @@ -585,54 +468,46 @@ }, "tag": { "type": "string", - "not": { - "const": "latest" - }, - "description": "Explicit floating-tag override for the spawned ingestor. EMPTY by default: the effective tag then comes from channelTags[CLIENT_ENV]. `latest` is rejected \u2014 spawn behaviour must not drift. To pin an exact image prefer `digest`, which wins over any tag. Surfaces as INGESTOR_IMAGE_TAG." - }, - "prodDigest": { - "type": "string", - "pattern": "^(sha256:[a-f0-9]{64})?$", - "description": "Prod reproducibility pin (sha256:<64 hex>), applied when `prodPin` resolves true (i.e. CLIENT_ENV resolves to prod). A CHART DEFAULT by design: `helm upgrade --reset-then-reuse-values` (the fleet auto-upgrade) adopts new chart defaults but re-applies stored user-supplied values, so only a default can update an installed edge (backend#1245). Must be a multi-arch index (#186); regenerate with scripts/resolve-ingestor-digest.sh --write and bump Chart.yaml." - }, - "prodPin": { - "type": "boolean", - "description": "Whether `prodDigest` applies on this edge. Default true = pin on prod only; dev/staging carry env.CLIENT_ENV=dev|stg and keep floating on `tag`. Set false on a canary prod edge to float it while the rest of the fleet stays pinned." - }, - "digest": { - "type": "string", - "pattern": "^(sha256:[a-f0-9]{64})?$", - "description": "Optional explicit per-edge pin (sha256:<64 hex>). Empty (default) = let prodPin/CLIENT_ENV decide. When set it wins over `prodDigest` and over `prodPin: false` in ANY environment, and every ingestion Job runs this exact image with imagePullPolicy=IfNotPresent (reproducibility). Must be a multi-arch index (#186). Surfaces as INGESTOR_IMAGE_DIGEST." + "not": { "const": "latest" }, + "description": "Explicit floating-tag override for the spawned ingestor. EMPTY by default: the effective tag then comes from channelTags[CLIENT_ENV] (backend#1360). `latest` is rejected — spawn behaviour must not drift. To pin an exact image prefer `digest`, which wins over any tag. Surfaces as INGESTOR_IMAGE_TAG." }, "channelTags": { "type": "object", - "description": "Per-environment floating tags, keyed on the resolved CLIENT_ENV, used when `tag` is empty. dev/stg track the unsigned internal channels published from the matching data-ingestors branch; prod stays a semver float (no :prod tag is published). backend#1360.", + "description": "Per-environment floating tags, keyed on the resolved CLIENT_ENV, used when `tag` is empty. `dev`/`stg` track the UNSIGNED internal channels published from the matching data-ingestors branch; `prod` stays a semver float because no :prod tag is published. backend#1360.", "properties": { "dev": { "type": "string", "minLength": 1, - "not": { - "const": "latest" - }, + "not": { "const": "latest" }, "description": "Tag spawned when CLIENT_ENV resolves to dev." }, "stg": { "type": "string", "minLength": 1, - "not": { - "const": "latest" - }, + "not": { "const": "latest" }, "description": "Tag spawned when CLIENT_ENV resolves to stg." }, "prod": { "type": "string", "minLength": 1, - "not": { - "const": "latest" - }, - "description": "Tag spawned when CLIENT_ENV resolves to prod." + "not": { "const": "latest" }, + "description": "Tag spawned when CLIENT_ENV resolves to prod and no digest applies." } } + }, + "prodDigest": { + "type": "string", + "pattern": "^(sha256:[a-f0-9]{64})?$", + "description": "Prod reproducibility pin (sha256:<64 hex>), applied when `prodPin` resolves true (i.e. CLIENT_ENV resolves to prod). A CHART DEFAULT by design: `helm upgrade --reset-then-reuse-values` (the fleet auto-upgrade) adopts new chart defaults but re-applies stored user-supplied values, so only a default can update an installed edge (backend#1245). Must be a multi-arch index (#186); regenerate with scripts/resolve-ingestor-digest.sh --write and bump Chart.yaml." + }, + "prodPin": { + "type": "boolean", + "description": "Whether `prodDigest` applies on this edge. Default true = pin on prod only; dev/staging carry env.CLIENT_ENV=dev|stg and keep floating on `tag`. Set false on a canary prod edge to float it while the rest of the fleet stays pinned." + }, + "digest": { + "type": "string", + "pattern": "^(sha256:[a-f0-9]{64})?$", + "description": "Optional explicit per-edge pin (sha256:<64 hex>). Empty (default) = let prodPin/CLIENT_ENV decide. When set it wins over `prodDigest` and over `prodPin: false` in ANY environment, and every ingestion Job runs this exact image with imagePullPolicy=IfNotPresent (reproducibility). Must be a multi-arch index (#186). Surfaces as INGESTOR_IMAGE_DIGEST." } } }, @@ -641,9 +516,7 @@ "properties": { "tag": { "type": "string", - "not": { - "const": "latest" - }, + "not": { "const": "latest" }, "description": "Image tag for mysql-client. Empty falls back to env.CLIENT_ENV. Must not be \"latest\"." }, "digest": { @@ -657,9 +530,7 @@ "properties": { "tag": { "type": "string", - "not": { - "const": "latest" - } + "not": { "const": "latest" } }, "digest": { "type": "string", @@ -679,28 +550,16 @@ "requests": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } }, "limits": { "type": "object", - "description": "CPU limit deliberately omitted in defaults \u2014 throttling causes InnoDB lock-wait timeouts.", + "description": "CPU limit deliberately omitted in defaults — throttling causes InnoDB lock-wait timeouts.", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } } } @@ -711,27 +570,15 @@ "requests": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } }, "limits": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } } } @@ -742,27 +589,15 @@ "requests": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } }, "limits": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } } } @@ -773,27 +608,15 @@ "requests": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } }, "limits": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } } } @@ -802,42 +625,21 @@ }, "priorityClass": { "type": "object", - "description": "Cluster-scoped PriorityClass for tracebloc data-plane workloads (mysql). An empty `name` disables the priorityClassName reference on the mysql pod entirely (mysql falls back to default priority 0 \u2014 only set this if you explicitly want no OOM protection from the scheduler).", + "description": "Cluster-scoped PriorityClass for tracebloc data-plane workloads (mysql). An empty `name` disables the priorityClassName reference on the mysql pod entirely (mysql falls back to default priority 0 — only set this if you explicitly want no OOM protection from the scheduler).", "properties": { - "create": { - "type": "boolean", - "default": true - }, - "name": { - "type": "string", - "default": "tracebloc-data-plane" - }, - "value": { - "type": "integer", - "minimum": 0, - "maximum": 1000000000, - "default": 1000000 - } + "create": { "type": "boolean", "default": true }, + "name": { "type": "string", "default": "tracebloc-data-plane" }, + "value": { "type": "integer", "minimum": 0, "maximum": 1000000000, "default": 1000000 } }, "allOf": [ { - "$comment": "If we are templating the PriorityClass resource, name must not be empty. `required: [create]` is needed because draft-07 `properties` only validates keys that are present \u2014 without it, omitting `create` passes the `if` vacuously and unconditionally enforces `then`. Same pattern as dockerRegistry below.", + "$comment": "If we are templating the PriorityClass resource, name must not be empty. `required: [create]` is needed because draft-07 `properties` only validates keys that are present — without it, omitting `create` passes the `if` vacuously and unconditionally enforces `then`. Same pattern as dockerRegistry below.", "if": { - "properties": { - "create": { - "const": true - } - }, - "required": [ - "create" - ] + "properties": { "create": { "const": true } }, + "required": ["create"] }, "then": { - "properties": { - "name": { - "minLength": 1 - } - } + "properties": { "name": { "minLength": 1 } } } } ] @@ -849,19 +651,13 @@ "mysql": { "type": "object", "properties": { - "create": { - "type": "boolean", - "default": true - } + "create": { "type": "boolean", "default": true } } }, "jobsManager": { "type": "object", "properties": { - "create": { - "type": "boolean", - "default": true - } + "create": { "type": "boolean", "default": true } } } } @@ -869,17 +665,13 @@ "clientId": { "type": "string", "minLength": 1, - "not": { - "pattern": "^<.*>$" - }, + "not": { "pattern": "^<.*>$" }, "description": "Client ID for authentication. Must be a real value, not a placeholder like ." }, "clientPassword": { "type": "string", "minLength": 1, - "not": { - "pattern": "^<.*>$" - }, + "not": { "pattern": "^<.*>$" }, "description": "Client authentication password. Must be a real value, not a placeholder like ." }, "autoUpgrade": { @@ -935,30 +727,18 @@ }, "image": { "type": "object", - "required": [ - "repository", - "tag" - ], + "required": ["repository", "tag"], "properties": { - "repository": { - "type": "string", - "minLength": 1 - }, + "repository": { "type": "string", "minLength": 1 }, "tag": { "type": "string", "minLength": 1, - "not": { - "const": "latest" - }, - "description": "Pin the helm version. `latest` is rejected \u2014 auto-upgrade behaviour must not drift." + "not": { "const": "latest" }, + "description": "Pin the helm version. `latest` is rejected — auto-upgrade behaviour must not drift." }, "pullPolicy": { "type": "string", - "enum": [ - "Always", - "IfNotPresent", - "Never" - ] + "enum": ["Always", "IfNotPresent", "Never"] } } }, @@ -968,27 +748,15 @@ "requests": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } }, "limits": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } } } @@ -1033,30 +801,18 @@ }, "image": { "type": "object", - "required": [ - "repository", - "tag" - ], + "required": ["repository", "tag"], "properties": { - "repository": { - "type": "string", - "minLength": 1 - }, + "repository": { "type": "string", "minLength": 1 }, "tag": { "type": "string", "minLength": 1, - "not": { - "const": "latest" - }, - "description": "Pin the alpine/k8s version. `latest` is rejected \u2014 image-refresh behaviour must not drift." + "not": { "const": "latest" }, + "description": "Pin the alpine/k8s version. `latest` is rejected — image-refresh behaviour must not drift." }, "pullPolicy": { "type": "string", - "enum": [ - "Always", - "IfNotPresent", - "Never" - ] + "enum": ["Always", "IfNotPresent", "Never"] } } }, @@ -1066,27 +822,15 @@ "requests": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } }, "limits": { "type": "object", "properties": { - "cpu": { - "type": "string", - "pattern": "^[0-9]+m?$" - }, - "memory": { - "type": "string", - "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" - } + "cpu": { "type": "string", "pattern": "^[0-9]+m?$" }, + "memory": { "type": "string", "pattern": "^[0-9]+(Ki|Mi|Gi|Ti)$" } } } } @@ -1094,10 +838,7 @@ } }, "dockerRegistry": { - "type": [ - "object", - "null" - ], + "type": ["object", "null"], "description": "Optional. Omit entirely or set null for public images (no secret or imagePullSecrets). Only create when set and create is true.", "properties": { "create": { @@ -1123,21 +864,12 @@ { "if": { "properties": { - "create": { - "const": true - } + "create": { "const": true } }, - "required": [ - "create" - ] + "required": ["create"] }, "then": { - "required": [ - "server", - "username", - "password", - "email" - ] + "required": ["server", "username", "password", "email"] } } ] From b87a4d49f8bb068e8bd2e3591fed262fbfa0bb5c Mon Sep 17 00:00:00 2001 From: LukasWodka Date: Thu, 30 Jul 2026 17:00:20 +0200 Subject: [PATCH 3/5] fix(review): one CLIENT_ENV normalizer, and unbreak the digest resolver Both from Bugbot on #494, and both caused by my own half-applied alias fix. 1. The alias normalization went into tracebloc.ingestorTag only, while tracebloc.ingestorDigest still compared the RAW CLIENT_ENV to "prod". So CLIENT_ENV=production got the prod float tag but an EMPTY digest -- silently dropping the reproducibility pin (backend#1028/#1245) on an edge that looked correctly configured, which is worse than the bug the alias fix was for. Extracted tracebloc.clientEnv as the single normalizer and pointed both helpers at it, so they cannot drift again (the reason ENV_ALIASES lives once in client-runtime proxy_config). 2. scripts/resolve-ingestor-digest.sh read images.ingestor.tag, now empty by default, so the documented no-arg / --write path exited on an empty tag -- the exact command the chart comments and the ingestor-multiarch CI error tell operators to run. It now falls back to images.ingestor.channelTags.prod on both the yq and the yq-free path, with a matching sibling awk reader scoped the same way. Verified: 311/311 helm unittest (a new case pins the production-alias digest); lint --strict clean on all four platform files; rendered tag+digest correct for prod/production/stg/staging/dev/development; the script's no-arg path resolves 0.7 -> the multi-arch digest again, and the portable reader returns 0.7 with tag empty. --- client/templates/_helpers.tpl | 40 +++++++++------- client/tests/ingestor_channel_tag_test.yaml | 16 +++++++ scripts/resolve-ingestor-digest.sh | 51 ++++++++++++++++++++- 3 files changed, 89 insertions(+), 18 deletions(-) diff --git a/client/templates/_helpers.tpl b/client/templates/_helpers.tpl index c7170664..d2020364 100644 --- a/client/templates/_helpers.tpl +++ b/client/templates/_helpers.tpl @@ -274,6 +274,28 @@ defeating the pin. Every read is nil-guarded for the same reason. Usage: {{ include "tracebloc.ingestorDigest" . }} */}} +{{/* + Resolved CLIENT_ENV, with the documented aliases normalized to the + canonical dev|stg|prod keys. + + ONE definition on purpose. Bugbot caught the first cut normalizing inside + tracebloc.ingestorTag only, so CLIENT_ENV=production selected the prod + float tag while tracebloc.ingestorDigest still compared the RAW value to + "prod" and returned nothing -- silently dropping the reproducibility pin + (backend#1028/#1245) on an edge that looked correctly configured. Any future + consumer of CLIENT_ENV must go through here rather than re-deriving it, the + same reason ENV_ALIASES lives once in client-runtime proxy_config. +*/}} +{{- define "tracebloc.clientEnv" -}} +{{- $raw := (default dict .Values.env).CLIENT_ENV | default "prod" -}} +{{- $aliases := dict "development" "dev" "staging" "stg" "production" "prod" -}} +{{- if hasKey $aliases $raw -}} +{{- get $aliases $raw -}} +{{- else -}} +{{- $raw -}} +{{- end -}} +{{- end }} + {{/* Effective floating tag for spawned ingestion Jobs (backend#1360). @@ -297,21 +319,7 @@ Usage: {{ include "tracebloc.ingestorDigest" . }} {{- if $explicit -}} {{- $explicit -}} {{- else -}} -{{- $clientEnv := (default dict .Values.env).CLIENT_ENV | default "prod" -}} -{{/* - Normalize the documented aliases before the lookup. The chart docs and the - SDK say dev|staging|prod while these keys are dev|stg|prod, and - client-runtime normalizes the same three at runtime - (proxy_config.ENV_ALIASES). Without this, CLIENT_ENV=staging -- the value - the schema documents -- missed channelTags entirely and fell back to the - prod float: the service would talk to the stg backend while spawning the - release ingestor, which is precisely the split-brain client-runtime#227 - was filed for. Review catch on client#494. -*/}} -{{- $aliases := dict "development" "dev" "staging" "stg" "production" "prod" -}} -{{- if hasKey $aliases $clientEnv -}} -{{- $clientEnv = get $aliases $clientEnv -}} -{{- end -}} +{{- $clientEnv := include "tracebloc.clientEnv" . -}} {{- $channels := default dict $ing.channelTags -}} {{- $channel := get $channels $clientEnv | default "" -}} {{- if $channel -}} @@ -332,7 +340,7 @@ Usage: {{ include "tracebloc.ingestorDigest" . }} {{- if hasKey $ing "prodPin" -}} {{- $prodPin = $ing.prodPin -}} {{- end -}} -{{- $clientEnv := (default dict .Values.env).CLIENT_ENV | default "prod" -}} +{{- $clientEnv := include "tracebloc.clientEnv" . -}} {{- if and $prodPin (eq $clientEnv "prod") -}} {{- $ing.prodDigest | default "" -}} {{- end -}} diff --git a/client/tests/ingestor_channel_tag_test.yaml b/client/tests/ingestor_channel_tag_test.yaml index baf7d667..bf26c94d 100644 --- a/client/tests/ingestor_channel_tag_test.yaml +++ b/client/tests/ingestor_channel_tag_test.yaml @@ -131,3 +131,19 @@ tests: content: name: INGESTOR_IMAGE_TAG value: "0.7" + + - it: the "production" alias keeps the prod reproducibility pin + # Bugbot catch (#494): the first alias fix normalized inside + # tracebloc.ingestorTag only, so CLIENT_ENV=production got the prod float + # tag while tracebloc.ingestorDigest still compared the RAW value to + # "prod" and returned nothing — silently dropping the pin + # (backend#1028/#1245) on an edge that looked correctly configured. Both + # helpers now share tracebloc.clientEnv. + set: + env.CLIENT_ENV: production + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: INGESTOR_IMAGE_DIGEST + value: "sha256:9098b3c9b83951f825e262d6ff1e9c47c46c5b5cafbfc8fef1ed82ccbadebb24" diff --git a/scripts/resolve-ingestor-digest.sh b/scripts/resolve-ingestor-digest.sh index ee416b2e..51252997 100755 --- a/scripts/resolve-ingestor-digest.sh +++ b/scripts/resolve-ingestor-digest.sh @@ -42,6 +42,38 @@ write=0 # Scoped to the images: -> ingestor: block so a sibling image's `tag:` # (jobsManager / podsMonitor / requestsProxy / ... each carry their own) # can never be picked up by mistake. bash-3.2 / macOS-safe (pure awk). +# Portable, yq-free reader for images.ingestor.channelTags.prod. Same scoping +# discipline as read_ingestor_tag: only the 6-space `prod:` leaf inside +# images: -> ingestor: -> channelTags: can match, so no sibling key can be +# picked up by mistake. bash-3.2 / macOS-safe. +read_ingestor_prod_channel() { + local file="$1" + [[ -f "$file" ]] || return 1 + awk ' + /^images:[[:space:]]*$/ { in_images = 1; next } + /^[^[:space:]#]/ { in_images = 0; in_ingestor = 0; in_channels = 0 } + in_images { + if ($0 ~ /^ [A-Za-z_][A-Za-z0-9_]*:[[:space:]]*$/) { + in_ingestor = ($0 ~ /^ ingestor:[[:space:]]*$/) ? 1 : 0 + in_channels = 0 + next + } + if (in_ingestor && $0 ~ /^ [A-Za-z_][A-Za-z0-9_]*:[[:space:]]*$/) { + in_channels = ($0 ~ /^ channelTags:[[:space:]]*$/) ? 1 : 0 + next + } + if (in_channels && $0 ~ /^ prod:[[:space:]]*/) { + line = $0 + sub(/^ prod:[[:space:]]*/, "", line) + gsub(/^["'"'"']|["'"'"']$/, "", line) + sub(/[[:space:]]*#.*$/, "", line) + gsub(/[[:space:]]*$/, "", line) + if (line != "") { print line; exit } + } + } + ' "$file" +} + read_ingestor_tag() { local file="$1" [[ -f "$file" ]] || return 1 @@ -79,15 +111,30 @@ if [[ -z "$tag" ]]; then # after the chart tag moves (e.g. 0.7 -> 0.8) while appearing to follow the # chart. Prefer yq; fall back to the portable yq-free parse above; if # neither can determine it, fail loudly rather than guess. + # Since backend#1360 `images.ingestor.tag` is an OVERRIDE that is empty by + # default, and the prod float lives in `images.ingestor.channelTags.prod`. + # This script resolves the PROD pin, so prefer the explicit override when an + # operator set one and otherwise read the prod channel. Without this the + # documented no-arg / --write path exits on an empty tag -- which is the very + # command the chart comments and the ingestor-multiarch CI error tell people + # to run (Bugbot, #494). if command -v yq >/dev/null 2>&1 && [[ -f "$chart_values" ]]; then tag="$(yq -r '.images.ingestor.tag' "$chart_values")" - [[ "$tag" == "null" ]] && tag="" # yq prints literal "null" for a missing key + [[ "$tag" == "null" ]] && tag="" + if [[ -z "$tag" ]]; then + tag="$(yq -r '.images.ingestor.channelTags.prod' "$chart_values")" + [[ "$tag" == "null" ]] && tag="" + fi else tag="$(read_ingestor_tag "$chart_values" || true)" + if [[ -z "$tag" ]]; then + tag="$(read_ingestor_prod_channel "$chart_values" || true)" + fi fi if [[ -z "$tag" ]]; then echo "ERROR: could not determine the default ingestor tag from ${chart_values#$here/../}." >&2 - echo " (images.ingestor.tag was unreadable: file missing, key absent, or yq not" >&2 + echo " (neither images.ingestor.tag nor images.ingestor.channelTags.prod was" >&2 + echo " readable: file missing, keys absent, or yq not" >&2 echo " installed and the yq-free parse found nothing.)" >&2 echo " Fix: pass TAG explicitly — scripts/resolve-ingestor-digest.sh [--write] —" >&2 echo " or install yq. Refusing to fall back to a hardcoded tag." >&2 From 8a5b82d2cdb9785aa10b1d99e12abec2c2de61c5 Mon Sep 17 00:00:00 2001 From: LukasWodka Date: Fri, 31 Jul 2026 08:59:53 +0200 Subject: [PATCH 4/5] style(script): keep each yq-free reader with its own header comment The new read_ingestor_prod_channel landed between read_ingestor_tag's header comment and read_ingestor_tag itself, so that header described the wrong function and read_ingestor_tag sat comment-less ~30 lines below. Moved the new function below the original; each header is now directly above the function it documents. No behaviour change -- both readers verified still returning 0.7 / empty, and the no-arg resolver path still resolves the multi-arch digest. --- scripts/resolve-ingestor-digest.sh | 60 +++++++++++++++--------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/scripts/resolve-ingestor-digest.sh b/scripts/resolve-ingestor-digest.sh index 51252997..fab5605b 100755 --- a/scripts/resolve-ingestor-digest.sh +++ b/scripts/resolve-ingestor-digest.sh @@ -42,6 +42,36 @@ write=0 # Scoped to the images: -> ingestor: block so a sibling image's `tag:` # (jobsManager / podsMonitor / requestsProxy / ... each carry their own) # can never be picked up by mistake. bash-3.2 / macOS-safe (pure awk). +read_ingestor_tag() { + local file="$1" + [[ -f "$file" ]] || return 1 + awk ' + # Enter the top-level images: block. + /^images:[[:space:]]*$/ { in_images = 1; next } + # Any other top-level key (col 0, not a comment) closes it. + /^[^[:space:]#]/ { in_images = 0; in_ingestor = 0 } + in_images { + # A 2-space sibling key under images: — arm the ingestor scope only + # while we are inside ingestor:, disarm on the next sibling. + if ($0 ~ /^ [A-Za-z_][A-Za-z0-9_]*:[[:space:]]*$/) { + in_ingestor = ($0 ~ /^ ingestor:[[:space:]]*$/) ? 1 : 0 + next + } + # The 4-space tag: leaf inside ingestor:. + if (in_ingestor && $0 ~ /^ tag:[[:space:]]/) { + v = $0 + sub(/^ tag:[[:space:]]*/, "", v) # drop the key + sub(/[[:space:]]+#.*$/, "", v) # drop a trailing comment + gsub(/^[[:space:]]+|[[:space:]]+$/, "", v) # trim + gsub(/^"|"$/, "", v) # unwrap double quotes + gsub(/^'\''|'\''$/, "", v) # unwrap single quotes + print v + exit + } + } + ' "$file" +} + # Portable, yq-free reader for images.ingestor.channelTags.prod. Same scoping # discipline as read_ingestor_tag: only the 6-space `prod:` leaf inside # images: -> ingestor: -> channelTags: can match, so no sibling key can be @@ -74,36 +104,6 @@ read_ingestor_prod_channel() { ' "$file" } -read_ingestor_tag() { - local file="$1" - [[ -f "$file" ]] || return 1 - awk ' - # Enter the top-level images: block. - /^images:[[:space:]]*$/ { in_images = 1; next } - # Any other top-level key (col 0, not a comment) closes it. - /^[^[:space:]#]/ { in_images = 0; in_ingestor = 0 } - in_images { - # A 2-space sibling key under images: — arm the ingestor scope only - # while we are inside ingestor:, disarm on the next sibling. - if ($0 ~ /^ [A-Za-z_][A-Za-z0-9_]*:[[:space:]]*$/) { - in_ingestor = ($0 ~ /^ ingestor:[[:space:]]*$/) ? 1 : 0 - next - } - # The 4-space tag: leaf inside ingestor:. - if (in_ingestor && $0 ~ /^ tag:[[:space:]]/) { - v = $0 - sub(/^ tag:[[:space:]]*/, "", v) # drop the key - sub(/[[:space:]]+#.*$/, "", v) # drop a trailing comment - gsub(/^[[:space:]]+|[[:space:]]+$/, "", v) # trim - gsub(/^"|"$/, "", v) # unwrap double quotes - gsub(/^'\''|'\''$/, "", v) # unwrap single quotes - print v - exit - } - } - ' "$file" -} - if [[ -z "$tag" ]]; then # No explicit TAG arg → default to the chart's images.ingestor.tag so this # helper always resolves the SAME line the chart ships. NEVER hardcode a From fa74943cb11a6385ce8e4a804fd7be9e39281f64 Mon Sep 17 00:00:00 2001 From: LukasWodka Date: Fri, 31 Jul 2026 09:05:47 +0200 Subject: [PATCH 5/5] fix: bump the chart to 1.9.9, and strip comments before quotes in the channel parser MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things, both caught by CI/review rather than by me: - The 'chart content ⇒ Chart.yaml version bump' guard was red: this PR edits client/templates and client/values.yaml, and a Helm repo publishes only on a version change, so without a bump the whole change would reach no install -- exactly how the perIngestionTables block shipped dark in #472. Chart 1.9.8 -> 1.9.9. - read_ingestor_prod_channel stripped quotes BEFORE removing an inline comment, the reverse of read_ingestor_tag. A channelTags.prod line with a trailing comment therefore parsed as 0.7" -- a stray quote -- and the no-arg/--write resolver would look up a nonexistent ref. Reordered to match: key, comment, trim, quotes. Verified across all four forms (double/single quoted with and without a trailing comment, and bare). --- client/Chart.yaml | 4 ++-- scripts/resolve-ingestor-digest.sh | 9 +++++---- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/client/Chart.yaml b/client/Chart.yaml index 5d3d77d6..af0b4c3e 100644 --- a/client/Chart.yaml +++ b/client/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: client description: A unified Helm chart for tracebloc on AKS, EKS, bare-metal, and OpenShift type: application -version: 1.9.8 -appVersion: "1.9.8" +version: 1.9.9 +appVersion: "1.9.9" keywords: - tracebloc - kubernetes diff --git a/scripts/resolve-ingestor-digest.sh b/scripts/resolve-ingestor-digest.sh index fab5605b..fa275f22 100755 --- a/scripts/resolve-ingestor-digest.sh +++ b/scripts/resolve-ingestor-digest.sh @@ -94,10 +94,11 @@ read_ingestor_prod_channel() { } if (in_channels && $0 ~ /^ prod:[[:space:]]*/) { line = $0 - sub(/^ prod:[[:space:]]*/, "", line) - gsub(/^["'"'"']|["'"'"']$/, "", line) - sub(/[[:space:]]*#.*$/, "", line) - gsub(/[[:space:]]*$/, "", line) + sub(/^ prod:[[:space:]]*/, "", line) # drop the key + sub(/[[:space:]]+#.*$/, "", line) # drop a trailing comment + gsub(/^[[:space:]]+|[[:space:]]+$/, "", line) # trim + gsub(/^"|"$/, "", line) # unwrap double quotes + gsub(/^'"'"'|'"'"'$/, "", line) # unwrap single quotes if (line != "") { print line; exit } } }