Skip to content

feat(cd): dispatch digest-pinned state updates to PostHog/charts after each release build - #136

Merged
fuziontech merged 1 commit into
mainfrom
sec/state-file-cd
Sep 3, 2026
Merged

feat(cd): dispatch digest-pinned state updates to PostHog/charts after each release build#136
fuziontech merged 1 commit into
mainfrom
sec/state-file-cd

Conversation

@fuziontech

Copy link
Copy Markdown
Member

Summary

Millpond deploys to the managed-warehouse clusters from the mutable :prod/:latest tags with pullPolicy: Always and an auto-restart CronJob. Anyone who can retag the GHCR package ships code to production within ~10 minutes with no PR, no approval, and no recorded provenance. A security review flagged this as a High finding.

This PR onboards millpond to the standard state-file digest-pin pipeline (like duckgres): after each release build, the workflow records <commit>@sha256:<digest> in state/millpond.yaml via commit_state_update repository-dispatch. ArgoCD rolls the pinned digest with prod-approval gating (require_prod_approval: true in the seeded state file).

Changes

  • release.yaml: github.repository == 'PostHog/millpond' guard; push a :<commit-sha> tag (digest inspection + traceability); compute the manifest digest; dispatch commit_state_update with release: "millpond" to PostHog/charts.
  • New actions pinned by SHA (same pins as duckgres's CD).
  • AGENT.md Releases section documents the dispatch.
  • No existing tags removed — :latest/:prod keep flowing until the charts-side cutover PR lands.

Prerequisites

  • GH_APP_CHARTS_DEPLOYER_APP_ID / GH_APP_CHARTS_DEPLOYER_PRIVATE_KEY secrets must exist in this repo (or org) — same app as duckgres uses. gh secret list shows neither; needs an org admin if not org-level.

Testing

  • YAML validated; dispatch payload mirrors the proven duckgres container-image-cache-proxy-cd.yml contract. First real validation is the next main build after merge: expect a state/millpond.yaml commit bumping image.sha.

Rollback

  • Revert; charts continues to follow the seed digest in state/millpond.yaml.

Adds the state-file CD step to the existing release workflow (no second
image pipeline): after the multi-arch image push, record the manifest
digest and dispatch commit_state_update to PostHog/charts so
state/millpond.yaml pins <commit>@sha256:<digest> and ArgoCD rolls it
with prod-approval gating.

Mutable :latest/:prod tags are kept for back-compat until the
charts-side cutover lands. Also tags the image with the commit SHA for
digest inspection and traceability, and guards the job to the canonical
repo.

Requires GH_APP_CHARTS_DEPLOYER_APP_ID / GH_APP_CHARTS_DEPLOYER_PRIVATE_KEY
secrets (not yet present in this repo).

Co-authored-by: Shelley <shelley@exe.dev>
@fuziontech
fuziontech requested a review from a team September 3, 2026 23:45
@jghoman

jghoman commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Workflow mechanics look right (fork guard, pinned actions, immutable :<sha> tag, digest from the raw manifest, back-compat tags retained, duckgres-shaped payload). Four cutover preconditions worth recording before the charts side lands:

  1. Seed order / frozen-pin trap: state/millpond.yaml doesn't exist in charts yet and no AppSet consumes it. Per charts' docs/claude/new-app-image-releases.md, the hand-seeded entry must be postdated by at least one real dispatch from this workflow or the pin freezes at seed forever. Merge this first so dispatches flow, and confirm what the charts dispatch handler does with an unknown release until the seed exists.
  2. Autobounce collision: the millpond fleets run autobounce CronJobs that rollout-restart on digest drift between the running image and the registry tag. Once deploys are digest-pinned, every release moves :latest/:prod while the pin holds — depending on the comparison logic that's either a permanent-drift restart loop or a silent no-op. The charts cutover must retire or gate autobounce in the same change.
  3. Promote semantics: this dispatches on every main merge, so "prod-approval gated" depends entirely on the charts workflow having the approval environment wired for this release — worth confirming in the companion rather than assuming.
  4. Consumer inventory: the millpond image also runs the megaduck + tenant maintenance CronJobs (viaduck namespace) and BOTH millpond elements (ws + replay) across three environments, plus autobounce itself. The pin cutover needs to reach all of them or the fleet ends up half-pinned.

@fuziontech
fuziontech merged commit 5c014a1 into main Sep 3, 2026
17 checks passed
@fuziontech
fuziontech deleted the sec/state-file-cd branch September 3, 2026 23:57
@fuziontech

Copy link
Copy Markdown
Member Author

Heads-up: the post-merge release build (run 33819790463) built and pushed the image fine, but the commit_state_update dispatch failed: GH_APP_CHARTS_DEPLOYER_APP_ID/GH_APP_CHARTS_DEPLOYER_PRIVATE_KEY are not available to this repo (the duckgres repo gets them via org-level secrets with selected-repository visibility). An org admin needs to grant this repo access to those org secrets (or add repo-level equivalents). Until then state/millpond.yaml stays at its seed digest — safe, but pins won't advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants