Skip to content

Phase E — Production wiring + staging validation + rollout/rollback runbook #100

Open
Open
Phase E — Production wiring + staging validation + rollout/rollback runbook#100
Labels
architectureStructural/system-level shape and runtime behaviourmajorMajor / load-bearing workpriority:p3Low - nice to haverequirements-targetTracked requirements-target item (joint-close)scope:estateAffects many or all repos across the estatestatus:blockedCannot proceed until a dependency clears

Description

@hyperpolymath
hyperpolymath
opened on May 17, 2026

Weeks 8–12. Wire tier-2 into BoJ; fill Trustfile.a2ml [CLOUDFLARE_EDGE_SECURITY].rate_limiting.tier_2_gateway PENDING; staging validation; rollout + rollback runbook. On landing: unblocks the unified core going public → unblocks K9 Dogfood green → idaptik#77 closable.

Parent: standards#91 (ADR-0004 phase E). Single-channel: do not work out of phase order; A→B→C→D→E.

Activity

  1. hyperpolymath
    added
    majorMajor / load-bearing work
    requirements-targetTracked requirements-target item (joint-close)
    on May 17, 2026
  2. hyperpolymath
    added a parent issue on May 17, 2026

  3. hyperpolymath commented on May 20, 2026

    @hyperpolymath
    hyperpolymath
    on May 20, 2026
    OwnerAuthor

    Phase E — first-session plan (2026-05-20)

    Kicking off Phase E. This is the long-pole that unblocks the unified core going public → K9 Dogfood green → idaptik#77 closable. Multi-session work over weeks; no code PR opens until this plan is on the issue.

    Live channel state (re-checked, not recalled)

    Phase E acceptance (restated for the record)

    1. Trustfile.a2ml [CLOUDFLARE_EDGE_SECURITY].rate_limiting.tier_2_gateway.status flipped from "PENDING — http-capability-gateway wiring forthcoming" (currently line 900 of .machine_readable/contractiles/trust/Trustfile.a2ml) to a real production value.
    2. HCG-in-front-of-BoJ wired in staging; staging health green under load that matches Phase D benchmark numbers.
    3. Rollout-and-rollback runbook lives in the BoJ repo at docs/runbooks/hcg-tier2-rollout.adoc (placement TBD — plan §E5 says docs/integration/gateway-rollback-runbook.md; I'll align on whichever exists / propose if neither).
    4. K9 Dogfood gate flips red → green in BoJ.
    5. HCG is the externally addressable surface; BoJ no longer reachable directly.

    Single-channel discipline — what Phase E can and cannot do now

    Phase D gates Phase E production rollout (plan §D "Blocks: Phase E production deployment requires D3 (regression alert) to be in place before rollout"). It does not gate Phase E scoping, runbook drafting, or audit of consumer-side assumptions. So this session will:

    • ✅ Drive — design/runbook/audit work that is risk-free w.r.t. Phase D state.
    • ❌ Hold — staging traffic-shift, production flip, and Trustfile.a2ml status: "DEPLOYED" flip until Phase D-3 (regression alert armed) and D-4 (real baseline numbers) land. Out-of-phase work is prohibited.

    First work-item: rollout/rollback runbook draft (E5-shaped)

    I'm proposing E5 (runbook) lands first because:

    • It captures the design while context is fresh and is the lowest-risk artefact in Phase E.
    • It's a pure-documentation PR — no code, no CI risk, no Phase D dependency.
    • It sets the operational vocabulary (traffic-shift mechanics, rollback triggers, dashboard names) that the later E2/E3/E4 PRs will reference.
    • Several details (on-call rota, prod cert material, dashboard URLs) need owner input — drafting now surfaces those gaps early so the owner can fill them async rather than blocking a code PR mid-rollout.

    PR will be Refs standards#100 and Refs standards#91 (never Closes — joint-close is owner-only).

    Second work-item (parallel-safe): consumer-side audit

    Before E1/E2 wiring PRs, I need a written catalogue of every BoJ site that currently assumes internal/loopback addressing. This is read-only and can be filed as a comment / draft note on #100 alongside the runbook PR. Outputs will include:

    • Cowboy bind addresses (current vs HCG-fronted target).
    • X-Trust-Level ingress paths in Elixir + Zig gnosis handler.
    • The Phase C finding: BojRest.TrustPolicy.satisfies?/3 does NOT enforce §3 invariant 3 (third clause matches regardless of is_local). Phase C documented this; Phase E must resolve it before BoJ is reachable from non-loopback. Owner-gated one-liner from the Phase C close-out: 
      def satisfies?(_required, _trust, false), do: false

    Deferred until Phase D-3 + D-4 land

    • E1 — Containerfile + k9-svc deployment spec finalisation (need confirmed runtime envelope from D-4).
    • E2 — staging deployment & traffic shift (needs D-3 gate armed, D-4 numbers to compare against).
    • E3 — telemetry verification under load (needs D-4 numbers to know what "good" looks like).
    • E4 — production flip (gated on E2 + E3 sign-off).

    What I will NOT do in this campaign

    • Touch the _wt-vclut-phase1 worktree or vcl-ut PR#21 (parallel session).
    • Touch standards#150–[#92 Class C] mongodb-mcp: missing BsonFieldType enum in Zig FFI #155 (Class C, parallel session).
    • Use --admin to bypass red checks; CI is the merge oracle.
    • Close idaptik#77 — owner-gated. Will surface "now closable" in a comment when K9 turns green.
    • Re-design HCG; Phase E is wiring an already-built thing.
    • Add scope; drift unrelated to Phase E gets a separate issue.

    Status comment cadence

    Mid-week status comment on this issue summarising what landed / what's blocked / what's next. Future-me and parallel sessions need breadcrumbs.

    Next action: draft the runbook PR.

    🤖 Generated with Claude Code

  4. hyperpolymath commented on May 20, 2026

    @hyperpolymath
    hyperpolymath
    on May 20, 2026
    OwnerAuthor

    Phase E first PR opened — boj-server#128 (E5 runbook draft)

    Filed as boj-server#128: docs(integration): HCG tier-2 rollout & rollback runbook (Phase E E5 draft) — DRAFT.

    hyperpolymath/boj-server#128

    Pure documentation. 308 lines covering prerequisites checklist, staging cut-over, 10/50/100% production rollout, observability, rollback (triggers + immediate-bypass + permanent-disable + post-decommission), and the final Trustfile flip. Refs standards#100 / #91, no Closes.

    Single-channel discipline preserved. §1.1 of the runbook explicitly gates execution on Phase D-3 (regression alert armed) and Phase D-4 (real baseline numbers populated). Until those land, this runbook is reviewed/merged-as-design only, not executed.

    Owner-input markers flag every field that needs operational context only the owner has — on-call rota, dashboard URLs, prod cert paths, traffic-shift mechanism choice, Cloudflare zone, freeze windows. Filing them now means the eventual rollout PR is not blocked mid-flight collecting answers.

    Phase C §3 invariant 3 status correction. Earlier comment on standards#91 said the BoJ-side fix in BojRest.TrustPolicy.satisfies?/3 was owner-gated. That comment was written before boj-server#106 (commit 40e46f6f) landed the deny clause — verified by git log against the file. Phase C is fully closed; Phase E inherits a clean §3 invariant.

    Next planned work-item: a written catalogue of every BoJ site that currently assumes internal/loopback addressing (input to E1/E2 design). Read-only audit, will be posted as a comment here rather than as a PR.

    🤖 Generated with Claude Code

  5. hyperpolymath commented on May 20, 2026

    @hyperpolymath
    hyperpolymath
    on May 20, 2026
    OwnerAuthor

    BoJ-side audit — loopback/internal addressing assumptions (input to E1/E2)

    Read-only audit of every site in boj-server that currently assumes internal/loopback addressing for BoJ. Filed as a comment rather than a PR per the first-session plan (this is information, not a change).

    Findings

    Already-compliant (no Phase E change needed)

    1. elixir/lib/boj_rest/router.ex lines 213–216 — loopback?/1 defines IPv4 (127.x.x.x) and IPv6 (::1) loopback. Used in /cartridge/:name/invoke (line 84) and /cartridge/:name/sse (line 136). Correctly threads is_local into check_trust and BojRest.CredentialDecryptor.extract/2.
    2. elixir/lib/boj_rest/trust_policy.ex line 73 — def satisfies?(_required, _trust, false), do: false already in force (boj-server#106, commit 40e46f6f). Non-loopback caller's X-Trust-Level is ignored.
    3. elixir/lib/boj_rest/credential_decryptor.ex lines 25–60 — plaintext credentials accepted only when is_local: true; non-loopback callers presenting plaintext credentials are rejected with an explicit "use encrypted form" reason. Same is_local flag flow as the router.
    4. Trustfile.a2ml [SEAMS] (Phase C, boj-server#90) — declares the gateway↔BoJ-gnosis seam with the loopback-only back-side bind invariant.
    5. elixir/test/credential_decryptor_test.exs + elixir/test/catalog_properties_test.exs + elixir/test/phase_c_seam_test.exs — 9-test seam coverage plus property tests on the loopback boundary. No gap on the test side.

    Pre-Phase-E exposed-bind risk surfaces (need confirmation before any §3.4 decommission)

    1. elixir/config/config.exs line 6 + elixir/lib/boj_rest/application.ex line 15 — Cowboy listens on port: 7700 with no explicit bind address. Plug.Cowboy defaults to :any (0.0.0.0), so without a Kubernetes NetworkPolicy / pod-network isolation the port is potentially externally addressable. Phase E §1.4 prereq is "BoJ staging instance addressable on loopback only" — this must be enforced at the container/k8s layer because the BEAM listener itself does not bind to 127.0.0.1 today. Action for E1/E2: either (a) add explicit ip: {127,0,0,1} to Plug.Cowboy.start_link options, or (b) document the NetworkPolicy requirement in k8s/deployment.yaml. ADR-0004 §1 (back-side bind isolation invariant) → contract §1 staging row says BoJ's :7700 is "not externally routable" — that is currently an operational assertion, not a code-enforced one.
    2. stapeln.toml line 100 — APP_HOST = "[::]" (all interfaces, IPv6). Same exposed-bind concern as (6). Used by the stapeln packager.
    3. k8s/deployment.yaml line 21 + k8s/service.yaml lines 10–22 — containerPort: 7700 exposed via Service on ports 7700 / 7701 / 7702 / 7703. Currently a ClusterIP/LoadBalancer (need to verify which) — if LoadBalancer the BoJ port is externally addressable today, defeating the §3.4 invariant before Phase E even starts. Action for E1/E2: convert the Service to a headless / ClusterIP-only address that the gateway pod can reach but external traffic cannot.
    4. gemini-extension.json, .mcp.json, smithery.yaml, README.md, k8s/deployment.yaml env — all default BOJ_URL=http://localhost:7700. These are MCP-bridge / CLI consumer configs, not production ingress, so they remain correct post-rollout (the gateway lives at a different address, MCP-bridge continues to talk to BoJ loopback directly via its local sidecar). No change needed but worth flagging the dual addressing in the Phase E telemetry verification (§E3): the gateway will not see MCP-bridge traffic, so dashboards should distinguish gateway-routed vs MCP-bridge-direct traffic.

    Adjacent — ADR-0013 streamable-HTTP transport (not Phase E scope but related)

    1. docs/decisions/0013-streamable-http-transport.md — defines BOJ_HTTP_AUTH modes (none / bearer / mtls / oidc). BOJ_HTTP_AUTH=none is documented as "loopback only; refuses non-loopback connections". This is a parallel (non-gateway) HTTP transport that already has a loopback-only mode. No Phase E change needed — but the documentation in §1.4 of the runbook should be cross-referenced so an operator doesn't accidentally enable BOJ_HTTP_AUTH=bearer on the :7700 listener and treat it as Phase-E-compliant external access (it isn't; the gateway is the only Phase-E-sanctioned external surface).

    Action items distilled (input to E1/E2 PRs, not to be done now)

    # Site Action Phase
    6 elixir/lib/boj_rest/application.ex:15 Add explicit ip: {127,0,0,1} to Plug.Cowboy options, OR document NetworkPolicy requirement E1
    7 stapeln.toml:100 Change APP_HOST to 127.0.0.1 for the Phase-E target deployment, OR document the container-isolation requirement E1
    8 k8s/deployment.yaml + k8s/service.yaml Convert BoJ Service to ClusterIP-only / pod-internal; ensure only the gateway pod can reach :7700 E1 + E2
    9 MCP-bridge configs No code change. Document the dual addressing in §E3 telemetry verification E3 (in runbook §2.2 already noted)

    Single-channel discipline reminder

    This audit is information only. The action items above are deferred until Phase D-3 + D-4 land (per the runbook §1.1 gate). When the gateway-side wiring PR is filed (E1/E2), it should reference this comment and pick up actions (6)/(7)/(8).

    🤖 Generated with Claude Code

  6. hyperpolymath
    added a commit that references this issue on May 20, 2026

    docs(integration): HCG tier-2 rollout & rollback runbook (Phase E E5 …

    92f3b3a

  7. hyperpolymath commented on May 20, 2026

    @hyperpolymath
    hyperpolymath
    on May 20, 2026
    OwnerAuthor

    Second Phase E PR opened — boj-server#130 (audit action #6, bind-tighten)

    Filed as boj-server#130: fix(boj): bind Cowboy to 127.0.0.1 by default (HCG tier-2 E1 prereq) — ready, not draft.

    hyperpolymath/boj-server#130

    Materialises audit action item #6 — Plug.Cowboy previously bound to 0.0.0.0 (Plug default) with no explicit ip: option, so the contract document's "BoJ's :7700 is not externally routable" claim and the Phase E runbook §1.4 prereq #6 were operational assertions, not code-enforced. This PR makes the default 127.0.0.1 with BOJ_BIND_IP override (failing fast on invalid input). 7 new unit tests cover the IP parser.

    All 18 CI checks green. Two pre-existing whisper-mcp cartridge property-test failures on mix test (catalog_properties_test.exs) are baseline rot on origin/main — verified by running tests against origin/main HEAD 92f3b3a1 in a clean detached worktree; same 2 failures, my +7 pass.

    Refs standards#100 / #91, no Closes. Independent of Phase D — the change is defensive hardening and ships now.

    Audit follow-ups still open (mentioned in PR body, deferred):

    Will pick these up next unless the owner wants a different next step.

    🤖 Generated with Claude Code

  8. hyperpolymath commented on May 20, 2026

    @hyperpolymath
    hyperpolymath
    on May 20, 2026
    OwnerAuthor

    Third Phase E PR opened — boj-server#131 (audit action #8, k8s ClusterIP)

    Filed as boj-server#131: fix(k8s): Service for BoJ to ClusterIP (HCG tier-2 E1 prereq) — DRAFT.

    hyperpolymath/boj-server#131

    Materialises audit action item #8 — k8s/service.yaml declared type: LoadBalancer, exposing all four BoJ ports externally. Defence-in-depth companion to PR#130 (Cowboy bind tightening). Together: BoJ binds loopback AND k8s Service is internal-only.

    Draft because this is breaking for anyone running the manifest as-is with a LoadBalancer in production. Owner gates merge on confirming no such deployment will break (header comment shows a kustomize/helm overlay for legacy/standalone needs).

    All 18 CI checks green. Estate cross-check: hypatia, rsr-certifier, opsm-service all use ClusterIP for backends — this PR brings BoJ in line. The only LoadBalancer in the estate is svalinn-gateway (a gateway, not a backend).

    Refs standards#100 / #91, no Closes.

    Phase E audit residue still open:

    • Action chore(deps): bump the actions group with 2 updates #7: stapeln.toml:100 APP_HOST = "[::]" → loopback (packager; separate concern).
    • Optional follow-up: NetworkPolicy for defence-in-depth beyond ClusterIP — not Phase E acceptance critical, can be filed as a tracked issue.

    Ready to pick up #7 next.

    🤖 Generated with Claude Code

  9. hyperpolymath
    added 2 commits that reference this issue on May 20, 2026

    fix(k8s): Service for BoJ to ClusterIP (HCG tier-2 E1 prereq) (#131)

    a3ae35f

    fix(boj): bind Cowboy to 127.0.0.1 by default (HCG tier-2 E1 prereq) (#…

    07d671c

  10. hyperpolymath commented on May 20, 2026

    @hyperpolymath
    hyperpolymath
    on May 20, 2026
    OwnerAuthor

    Fourth Phase E PR opened — boj-server#132 (audit action #7, container APP_HOST)

    Filed as boj-server#132: fix(container): APP_HOST defaults to 127.0.0.1 (HCG tier-2 E1 prereq) — DRAFT.

    hyperpolymath/boj-server#132

    Materialises audit action item #7. Scope expanded during implementation from one site (the audit named stapeln.toml) to three sites — entrypoint.sh × 2 and compose.prod.yaml had the same [::] default that the audit missed. All three feed the same Zig-adapter --host flag, so they need to flip together.

    Now the consumer-side audit is fully addressed across the three companion PRs:

    Audit # Layer PR State
    #6 Elixir Cowboy bind boj-server#130 open, 18 checks green
    #8 k8s Service type boj-server#131 DRAFT, 18 checks green
    #7 Zig adapter APP_HOST defaults boj-server#132 DRAFT, CI didn't auto-trigger (see PR comment) — manual Governance dispatch green

    Defence in depth achieved: Elixir Cowboy binds loopback (#130) AND Zig adapter binds loopback (#132) AND k8s Service is internal-only (#131). Three independent layers, any one of which would mitigate the §3.4 invariant violation.

    CI anomaly on #132: pull_request workflows did not fire on the branch. Other PRs in this session (#130, #131) triggered normally. Likely cause: estate-wide CI concurrency-pool exhaustion from the active sweep122 campaign (Terminal-7 PID 244005). Manual workflow_dispatch of Governance returned all 6 jobs green. Documented in PR#132 comment.

    Phase E audit-residue close-out: the only optional follow-up still open is a NetworkPolicy for defence-in-depth beyond ClusterIP. Not Phase E acceptance critical. Can file as a tracked issue if you want.

    Refs standards#100 / #91, no Closes.

    🤖 Generated with Claude Code

  11. hyperpolymath commented on May 20, 2026

    @hyperpolymath
    hyperpolymath
    on May 20, 2026
    OwnerAuthor

    Optional follow-up filed — boj-server#135 (NetworkPolicy)

    Filed boj-server#135 for the optional NetworkPolicy defence-in-depth layer noted in the consumer-side audit. Not Phase E acceptance critical; priority Low. Captures the proposed manifest shape, threat models it covers beyond ClusterIP, override path for non-HCG-fronted deployments, and CNI-support caveats. Picks up after Phase E exit unless an incident raises priority.

    Phase E audit residue is now fully tracked.

    🤖 Generated with Claude Code

  12. hyperpolymath
    added 2 commits that reference this issue on May 20, 2026

    fix(container): APP_HOST defaults to 127.0.0.1 (HCG tier-2 E1 prereq)

    8a0fca2

    fix(container): APP_HOST defaults to 127.0.0.1 (HCG tier-2 E1 prereq) (…

    5699de0

  13. 10 remaining items

  14. hyperpolymath
    added 9 commits that reference this issue on Jun 14, 2026

    feat(scripts): hcg-policy-smoke.sh — unknown-path default-deny canary…

    c710e25

    feat(scripts): hcg-policy-smoke.sh — stealth-profile status canary (P…

    ae7532a

    docs(integration): Phase E observability spec (standards#100)

    c959198

    docs(integration): Phase E observability spec (standards#100) (#226)

    711201d

    feat(hcg): add surface-drift check script (standards#100 §1.5)

    feaf2d3

    feat(hcg): add surface-drift check script (standards#100 §1.5) (#228)

    64a70c5

    ci(hcg): wire surface-drift check into Actions (standards#100 §1.5) (#…

    37f6635

    feat(hcg): add spec-coverage check script (standards#100 §1.5) (#230)

    567f244

    docs(integration): Phase E §1.4 prereq — BoJ-side observability spec …

    7e134bc

  15. hyperpolymath commented on Aug 27, 2026

    @hyperpolymath
    hyperpolymath
    on Aug 27, 2026
    OwnerAuthor

    Next step (re-verified 2026-08-27): precisely scoped and untouched — flip boj-server Trustfile line 900 tier_2_gateway.status from PENDING, and write the rollback runbook (docs/DEPLOYMENT.adoc and SCOPED-DEPLOYMENT.adoc exist; rollback does not).

    → Gated on #91's artifacts actually passing — presence was verified, green was not. Run the mTLS/E2E/bench suites first; if green, this is small. The staging-validation half is the real cost.

  16. hyperpolymath
    added a commit that references this issue on Sep 22, 2026

    fix(ci): codeql-action v4.38.1 -> v4.38.0 SHA pin + dependabot hold (…

    2eb9bb1
  17. hyperpolymath
    added
    architectureStructural/system-level shape and runtime behaviour
    priority:p3Low - nice to have
    scope:estateAffects many or all repos across the estate
    status:blockedCannot proceed until a dependency clears
    on Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureStructural/system-level shape and runtime behaviourmajorMajor / load-bearing workpriority:p3Low - nice to haverequirements-targetTracked requirements-target item (joint-close)scope:estateAffects many or all repos across the estatestatus:blockedCannot proceed until a dependency clears

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions