adaptive_export: reliable dx-steered pem-direct capture (chunk/end_time, breaker, dc_snoop filter, DaemonSet)

bazel/repository_locations.bzl

@@ -456,7 +456,10 @@ REPOSITORY_LOCATIONS = dict(
     org_libc_musl = dict(
         sha256 = "7d5b0b6062521e4627e099e4c9dc8248d32a30285e959b7eecaa780cf8cfd4a4",
         strip_prefix = "musl-1.2.3",
-        urls = ["http://musl.libc.org/releases/musl-1.2.3.tar.gz"],
+        urls = [
+            "https://sources.openwrt.org/musl-1.2.3.tar.gz",
+            "http://musl.libc.org/releases/musl-1.2.3.tar.gz",
+        ],
         manual_license_name = "libc/musl",
     ),
     rules_cc = dict(

ci/artifact_utils.sh

@@ -107,7 +107,15 @@ create_manifest_update() {
   tag_name="release/${component}/v${version}"
   # actions/checkout doesn't get the tag annotation properly.
   git fetch origin tag "${tag_name}" -f
-  timestamp="$(git tag -l --format "%(taggerdate:raw)" "${tag_name}"  | awk '{print $1}' | jq '. | todate')"
+  # taggerdate is empty for a LIGHTWEIGHT tag → produces `timestamp: ,` → jq syntax
+  # error → release-metadata step fails even though the image built fine. Fall back to
+  # the tagged commit's committer date so the manifest is well-formed regardless of how
+  # the release tag was cut (annotated vs lightweight).
+  raw_ts="$(git tag -l --format "%(taggerdate:raw)" "${tag_name}" | awk '{print $1}')"
+  if [ -z "${raw_ts}" ]; then
+    raw_ts="$(git log -1 --format="%ct" "${tag_name}")"
+  fi
+  timestamp="$(printf '%s' "${raw_ts}" | jq '. | todate')"
 
   jq -s \
     "[{name: \"${component}\", artifact: [{timestamp: ${timestamp}, commitHash: \"${commit_hash}\", versionStr: \"${version}\", availableArtifactMirrors: .}]}]" \

k8s/vizier/BUILD.bazel

@@ -23,7 +23,6 @@ load("//bazel:kustomize.bzl", "kustomize_build")
 package(default_visibility = ["//visibility:public"])
 
 VIZIER_IMAGE_TO_LABEL = {
-    "$(IMAGE_PREFIX)/vizier-adaptive_export_image:$(BUNDLE_VERSION)": "//src/vizier/services/adaptive_export:adaptive_export_image",
     "$(IMAGE_PREFIX)/vizier-cert_provisioner_image:$(BUNDLE_VERSION)": "//src/utils/cert_provisioner:cert_provisioner_image",
     "$(IMAGE_PREFIX)/vizier-cloud_connector_server_image:$(BUNDLE_VERSION)": "//src/vizier/services/cloud_connector:cloud_connector_server_image",
     "$(IMAGE_PREFIX)/vizier-kelvin_image:$(BUNDLE_VERSION)": "//src/vizier/services/agent/kelvin:kelvin_image",

k8s/vizier/adaptive_export/README.md

@@ -0,0 +1,58 @@
+# The export path: what is defined where
+
+Two fork-only components move query results off a cluster: `adaptive-export` in the `pl`
+namespace, and `dx-daemon` in the `honey` namespace. This file is the map, so nothing has
+to be remembered twice.
+
+## One installer
+
+```
+skaffold deploy -f skaffold/skaffold_export.yaml
+```
+
+Two things have to exist first: a running Vizier, and the ClickHouse that holds
+`forensic_db`. adaptive-export applies its own schema at startup and exits rather than run
+against tables it cannot verify, so without ClickHouse it installs correctly and then
+crashloops on `schema apply failed; refusing to proceed with possibly drifted tables`. That
+is adaptive-export being careful, not a broken install. Install ClickHouse, then this.
+
+That is the whole install, on an empty cluster or on top of a running one. It is
+idempotent: a second run with no change touches nothing and restarts nothing. A pod is
+replaced only when an image pin changes. The file holds two Skaffold configs,
+`adaptive-export` and `dx-daemon`, and `dx-daemon` declares `requires: adaptive-export`,
+so the order is declared rather than remembered. Select one with `-m <config>`.
+
+## Where each thing lives
+
+| Thing | Defined in |
+| --- | --- |
+| adaptive-export DaemonSet and Service | `k8s/vizier/adaptive_export/daemonset.yaml` |
+| adaptive-export ServiceAccount, Role, ClusterRole and bindings | `k8s/vizier/adaptive_export/rbac.yaml` |
+| adaptive-export image pin | `images:` in `k8s/vizier/adaptive_export/kustomization.yaml` |
+| dx-daemon DaemonSet | `k8s/vizier/dx/dx-daemon.yaml` |
+| dx-daemon image pin | `images:` in `k8s/vizier/dx/kustomization.yaml` |
+| every secret and ConfigMap fix-up | the `before` hooks in `skaffold/skaffold_export.yaml` |
+
+Each component is self-contained. The AE kustomization does not reach into `../bootstrap`,
+which is what used to render the same two files twice under two different label sets and
+leave `adaptive-export-control` with no endpoints.
+
+## Not part of a vizier release
+
+The release bundle is built by the `kustomize_build` targets in `k8s/vizier/BUILD.bazel`,
+which glob `base/`, `bootstrap/`, `pem/` and the two metadata overlays. Neither
+`adaptive_export/` nor `dx/` is in any of those globs, so a vizier release installs
+neither, and this path is rolled on its own with the command above.
+
+## Inputs the installer reads
+
+| Variable | Used for | If unset |
+| --- | --- | --- |
+| `PIXIE_API_KEY`, or `PX_API_KEY` | the `pixie-api-key` in `pl-adaptive-export-secrets` | a live secret is left alone; on a cluster with no secret the install stops |
+| `AE_CH_DSN` | the `clickhouse-dsn` in the same secret | the in-cluster forensic database default |
+| `AE_PULL_CONFIG`, or `DOCKER_CONFIG` | the `adaptive-export-pull` image pull secret | an existing pull secret is reused; otherwise the install stops |
+| `DX_PULL_CONFIG`, or `DOCKER_CONFIG` | the `dx-pull` image pull secret | a default config location is tried, and the install stops if nothing there is readable |
+| `DX_CH_HTTP_URL` | dx's database endpoint | the in-cluster default |
+
+Secrets are only ever seeded, never overwritten with a placeholder. A key is written only
+when one is supplied.

k8s/vizier/adaptive_export/daemonset.yaml

@@ -0,0 +1,109 @@
+---
+# adaptive-export: node-local forensic capture operator. DaemonSet so each pod
+# queries its own node's vizier-pem (pem-direct). Secret seeded per-cluster.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: adaptive-export
+  labels: {name: adaptive-export, plane: control}
+spec:
+  selector:
+    matchLabels: {name: adaptive-export}
+  template:
+    metadata:
+      labels: {name: adaptive-export, plane: control}
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - {key: kubernetes.io/os, operator: In, values: [linux]}
+      serviceAccountName: pl-adaptive-export-service-account
+      imagePullSecrets:
+      - name: adaptive-export-pull
+      containers:
+      - name: adaptive-export
+        image: vizier-adaptive_export_image:latest
+        ports:
+        - {name: control, containerPort: 9100, hostPort: 9100}
+        resources:
+          requests: {cpu: 100m, memory: 128Mi}
+          limits: {cpu: "1", memory: 1Gi}
+        env:
+        - name: HOST_IP
+          valueFrom: {fieldRef: {fieldPath: status.hostIP}}
+        # Profiling listener, bound to loopback: unreachable from the cluster,
+        # reachable through `kubectl port-forward` for a heap or CPU sample.
+        - {name: AE_PPROF_ADDR, value: "127.0.0.1:6061"}
+        - name: ADAPTIVE_VIZIER_DIRECT_ADDR
+          value: "$(HOST_IP):50305"
+        - name: PL_JWT_SIGNING_KEY
+          valueFrom: {secretKeyRef: {name: pl-cluster-secrets, key: jwt-signing-key}}
+        - name: PX_DISABLE_TLS
+          value: "1"
+        - name: PL_NAMESPACE
+          valueFrom: {fieldRef: {fieldPath: metadata.namespace}}
+        - name: NODE_NAME
+          valueFrom: {fieldRef: {fieldPath: spec.nodeName}}
+        - name: PIXIE_API_KEY
+          valueFrom: {secretKeyRef: {name: pl-adaptive-export-secrets, key: pixie-api-key}}
+        - name: CLICKHOUSE_DSN
+          valueFrom: {secretKeyRef: {name: pl-adaptive-export-secrets, key: clickhouse-dsn}}
+        - name: KUBESCAPE_TABLE
+          value: "kubescape_logs"
+        - name: EXPORT_MODE
+          value: "never"
+        # Control surface is secure-by-default (#96): TLS + bearer-JWT auth are ON
+        # out of the box. The service-tls-certs keypair mounted at /certs below is
+        # used for TLS (else AE self-generates an ephemeral in-memory cert), and
+        # PL_JWT_SIGNING_KEY above turns on auth. CONTROL_TLS / CONTROL_REQUIRE_AUTH
+        # are deprecated no-ops; set CONTROL_INSECURE=true only to opt out (dev).
+        - name: CONTROL_ADDR
+          value: ":9100"
+        - name: ADAPTIVE_PUSH_PIXIE_ROWS
+          value: "true"
+        # Capture is driven by the control surface only; no kubescape_logs poll.
+        - name: ADAPTIVE_TRIGGER_POLL
+          value: "false"
+        # Convert any evidence table still on a plain MergeTree. Leader-only,
+        # and a no-op once every table has been converted.
+        - name: ADAPTIVE_MIGRATE_ENGINES
+          value: "true"
+        - name: ADAPTIVE_RECONCILE
+          value: "true"
+        - name: DEPLOY_TRACEPOINTS
+          value: "true"
+        - name: INSTALL_PRESET_SCRIPTS
+          value: "false"
+        - name: ADAPTIVE_MAX_INFLIGHT_QUERIES_GLOBAL
+          value: "4"
+        - name: ADAPTIVE_ORDER_CHUNK_SEC
+          value: "600"
+        - name: VERBOSE
+          value: "true"
+        volumeMounts:
+        - {name: certs, mountPath: /certs, readOnly: true}
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities: {drop: [ALL]}
+          seccompProfile: {type: RuntimeDefault}
+      volumes:
+      - name: certs
+        secret: {secretName: service-tls-certs}
+      securityContext:
+        runAsUser: 10100
+        runAsGroup: 10100
+        fsGroup: 10100
+        runAsNonRoot: true
+        seccompProfile: {type: RuntimeDefault}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: adaptive-export-control
+spec:
+  selector: {name: adaptive-export}
+  internalTrafficPolicy: Local   # dx reaches its co-located (same-node) AE
+  ports:
+  - {name: control, port: 9100, targetPort: 9100}

k8s/vizier/adaptive_export/kustomization.yaml

@@ -0,0 +1,23 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: pl
+# Everything adaptive-export needs lives in this directory. It used to reach back into
+# ../bootstrap, which meant the same two files were rendered by two kustomizations: this
+# one and the vizier bootstrap. The two applied different commonLabels, so the Service
+# selector written by one did not match the pods written by the other, and a deploy on top
+# of a vizier-installed AE left adaptive-export-control with no endpoints at all — dx's
+# hand-off silently going nowhere. One definition, one label set, no divergence possible.
+# The pl-adaptive-export-secrets secret is NOT a resource here. It is created by the
+# before-deploy hook in skaffold/skaffold_export.yaml, which seeds it only when a key is
+# supplied and otherwise leaves a live secret untouched. There used to be a second
+# definition of the same secret in this directory (secrets.example.yaml); whenever it was
+# applied it clobbered the real pixie-api-key with its placeholder, which is the recurring
+# "AE unauthenticated / writes 0" bug. One definition, in the installer. See README.md.
+resources:
+- rbac.yaml
+- daemonset.yaml
+images:
+- name: vizier-adaptive_export_image
+  newName: docker.io/tanzeee/adaptive-export
+  newTag: 0.16.50-6dc818c

k8s/vizier/bootstrap/adaptive_export_role.yaml → k8s/vizier/adaptive_export/rbac.yaml

@@ -37,6 +37,22 @@ subjects:
   name: pl-adaptive-export-service-account
   namespace: pl
 ---
+# The ONE cluster-scoped grant adaptive-export still needs, and why.
+#
+# config.findVizierNamespace() lists pods across the cluster to locate the vizier
+# namespace, then reads pl-cluster-secrets there for cluster-id and cluster-name. Without
+# it AE logs "unable to find vizier namespace: pods is forbidden ... at the cluster scope",
+# falls back to environment variables, and fatals on a missing PIXIE_CLUSTER_ID. Verified
+# by removing it: the pods CrashLoop. main.go's leader election lists pods too, but only in
+# its own namespace, so this covers both and the Role above does not repeat it.
+#
+# What this role no longer grants, because it was dead: read on the kubescape
+# containerprofiles and rogueartifacts CRDs and on the node-agent-bundle-policy ConfigMap.
+# Those belonged to kssync, which has moved out of adaptive-export and carries its own
+# RBAC where it now lives.
+#
+# To retire this ClusterRole entirely, AE would have to take the vizier namespace from
+# PL_NAMESPACE — already injected into its pod spec — instead of discovering it.
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata: