chore: adopt production as the gated release source (closes #152) - #194
Conversation
|
@schutera — this PR is a policy/governance change and needs your explicit approval, not just a code review. The decision you're approvingFrom now on, all production releases — web services and firmware OTA — ship from the If you'd rather keep deploying from Two operator steps that are yours (can't be done from this PR)These need push access to
Status of the PR itself
🤖 Generated with Claude Code |
Reconcile the documented services deploy source with reality and unify it with firmware OTA on a single gated `production` branch. Investigation for #152 found three stacked problems: the docs named `production` while the live auto-deploy pulled `main`; firmware OTA and the services track were documented as separate; and `main`/`production` shared no common git ancestor (main's history was rebuilt), so `production` could never fast-forward and silently rotted. Decision (per maintainer): `production` becomes the single gated release branch for both web services and firmware OTA. `main` is the integration line; a release is a fast-forward of `production` onto a chosen `main` commit. `prod-*` tags are cut on `production`. - scripts/deploy.sh: BRANCH main -> production; branch-agnostic notify text - production-deployment.md: drop drift warning; add release/promotion + one-time host cutover section - production-runbook.md: document the promote-then-pull model - firmware-release.md: rewrite the branch & tag model (both tracks on production); replace the "known drift" callout with a history note; update the release-checklist commit/tag step - chapter 11: mark the drift lesson RESOLVED; record the unrelated-history root cause and the fast-forwardable-deploy-branch rule - new ADR-028; update README/esp-flashing/CLAUDE.md pointers The branch reconciliation (archive tag + force-reset of origin/production) and the one-time prod-host checkout are operator steps documented in ADR-028 and production-deployment.md, to run after this lands on main. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017drgAN84qrn61eZ1yZTdgS
The production-as-gated-release-branch model (#152 / ADR-030) was only spelled out in the firmware-OTA section. Add a concise hard rule to the top-level "Critical rules (do NOT violate)" list so every session knows prod releases ship from `production`, never from `main`. Links to the full mechanics rather than duplicating them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
c0981d1 to
ebfde09
Compare
…loud Three conflicts, all where #196's deploy.sh hardening met this branch's BRANCH=main -> production switch. Resolved by keeping BOTH sides rather than picking one: - deploy.sh header: kept main's dependency-install CAVEAT (npm ci wipes node_modules in place, pip upgrades are forward-only and not reverted on rollback) and applied this branch's production wording. - deploy.sh config block: kept FAILED_MARKER (the retry-storm guard) and set BRANCH="production". - .deploy.env.example: kept the new pm2-environment section (HIGHFIVE_ENV / HIGHFIVE_API_KEY precedence) and this branch's "deploy branch (production)" wording on the firmware-OTA note. - production-runbook.md: kept the gated-release-branch explanation AND main's simplified-hand-deploy caveats, and corrected the manual step from `git pull --ff-only origin main` to `production` — it would otherwise have told an operator to hand-deploy from the integration line this change exists to stop deploying from. Beyond the merge, one behavioural addition, because this switch has a failure mode nobody would see. deploy.sh exits early when the host checkout is not on BRANCH: [ "$cur_branch" != "$BRANCH" ] && log "skip"; exit 0 That was harmless while BRANCH was `main` (the host is on main). Flipping it to `production` makes it fire on EVERY tick until someone runs `git checkout production` on the host -- so the moment this merges, auto-deploy stops, silently, with no notification, while `main` keeps accumulating commits nobody realises are unshipped. A silent stop is the worst shape for this: it looks exactly like "no new commits". It now sends ONE Discord notification naming the exact cutover command, with a marker file so it doesn't repeat every two minutes, and clears the marker once the host is on the right branch. Verified: bash -n clean, shellcheck -S info clean, the 11-case rollback harness and 13-case gate matrix from #196 still pass, and the doc-citation, bind-claims and hardcoded-key gates are green. Co-Authored-By: WOZCODE <contact@withwoz.com>
Senior review of the production-branch adoption. Two P0s, both verified with single git commands against the repo rather than read off the prose. 1. THE DOCUMENTED CUTOVER WOULD HAVE REVERTED PRODUCTION. The procedure said "git checkout production && git reset --hard origin/production" with no promotion step. But origin/production is a0e7374, four commits behind main: git show origin/production:scripts/deploy.sh | grep '^BRANCH=' -> "main" ...and 0 hits for FAILED_MARKER|NPM_CI_RAN|PIP_FAILED|add_reload| rollback_health_targets|HUSKY=0 (26 on this branch) So running it today rolls the live host back past #193, #196 and #222 -- discarding the deploy hardening and the security track-A work -- and leaves the checkout on `production` running a driver that tracks `main`. That takes the old bare `log "skip"; exit 0` path: deploys stop silently, forever, and the wrong-branch alert that would have said so was reverted away with everything else. Silent, permanent, self-concealing. The procedure is now explicitly ordered: promote, VERIFY (`git show origin/production:scripts/deploy.sh | grep '^BRANCH='` must print production), then check out. The same ordering and the same verify command are in the Discord alert body, because that is what the operator actually reads at 2am. 2. ADR-030's recorded root cause was FALSE, and it was the sole justification for force-resetting a release branch: git rev-list --max-parents=0 main -> d9ac93d (one root) git rev-list --max-parents=0 bf8b314 -> d9ac93d (the SAME root) git merge-base main bf8b314 -> da1b21d git rev-list --count main..bf8b314 -> 25 (not 136) There was no orphan root, no rebuilt history, no unrelated ancestry -- a merge was available the entire time and was rejected for tidiness. The cited #124 is a senior-reviewer config commit. Corrected in ADR-030 and in chapter 11, kept as a visible correction rather than a quiet edit, because ch11 had already generalised the false cause into a "how to avoid this next time" rule that future maintainers would have applied to a scenario that never happened. Also from this round: - "A real promotion gate" was overstated: neither branch has protection, CI runs on main only, and any fast-forwarding commit is accepted. Now says gate-by-convention, and states that production MUST stay unprotected -- protecting it would reject publish_firmware's push and ship OTAs whose SEQUENCE bump is not in git. - The ADR's one acknowledged invariant violation (publish_firmware commits to production) had "tracked as a follow-up" with nothing behind it. Filed #225. - The OTA notification -- attached to the single irreversible action in the system -- cited ADR-028 (ML inference server-side) instead of ADR-030. This is the ADR-renumber-on-collision trap the repo already documented. - The archive tag is NOT on the remote (`git ls-remote --tags origin` finds nothing), so the 25 commits survive only via a stale branch clean_gone would delete. Both docs now say so instead of claiming a recovery point exists. - CONTRIBUTING.md, which CLAUDE.md names as the authority for the branch model, said only "branch off main" and never mentioned production. It now documents the promote-don't-PR rule and the never-force-push constraint. - The branch-mismatch marker is hoisted to a BRANCH_MARKER constant next to FAILED_MARKER (it was a local var re-typed as a literal in the rm), and the marker is now written BEFORE notify -- notify can fail under set -e, which would have produced the every-two-minutes alert the marker exists to stop. Not done, stated plainly: image-service/tests/test_upload.py carries ruff format reflow. main's version is not ruff-clean, and the pre-commit hook reformats any staged .py, so it cannot be reverted without bypassing the hook. Verified: bash -n and shellcheck -S info clean; #196's 11-case rollback harness and 13-case gate matrix still pass; all five repo gates green. Co-Authored-By: WOZCODE <contact@withwoz.com>
…over honestly Round-2 review. The pattern in the finding is the same one this PR is about. 1. The false "shared no common ancestor" claim was fixed in ADR-030 and chapter 11 and LEFT STANDING in docs/07-deployment-view/firmware-release.md -- so the branch shipped three documents describing one event, two of which called the third a fabrication. The survivor was the runbook: the file someone opens WHILE cutting a release, while the retractions sat in an ADR and a tech-debt log nobody reads mid-release. Fixed, with the retraction visible there too. Swept the tree afterwards; the only remaining matches are inside the correction blocks that quote the claim in order to retract it. 2. "verified stale" did not survive its own standard. The ADR had just spent twenty lines explaining why unverified assertions here are dangerous, and then rested the whole justification for discarding 25 commits on an enumeration that omitted four test files and never mentioned the ESP work. Re-derived from the repo, and one finding is worth the trouble: the archived TIP (bf8b314, "use esp_task_wdt_reconfigure and defer loopTask subscribe past AP setup") uses an API that appears NOWHERE on main -- `git grep esp_task_wdt_reconfigure origin/main -- ESP32-CAM/` is empty, and main still uses the IDF-4 esp_task_wdt_init/add pair. Main fixes the same AP-mode reboot loop a different way (>=60s TASK_WDT_TIMEOUT_S plus runAccessPoint feeding the watchdog, recorded as fixed in troubleshooting.md), so nothing live is lost -- but "already exists on main" was the wrong description, and the ADR now says what was actually checked. 3. The OTA notification told the operator to run `git checkout main` with no statement of WHERE. It arrives while they are looking at the host, and the commands run there -- where `git checkout main` immediately trips the branch-mismatch guard this same PR adds and pauses every deploy. Now says "FROM A MAINTAINER CLONE, NOT THIS HOST" and explains the consequence. 4. The reordered cutover fixed the direction but not the rebuild. `git reset --hard` restores the source tree only; backend/dist, homepage/dist and node_modules stay at whatever the host last built, and no later tick repairs that because deploy.sh exits at `[ "$PREV_SHA" = "$REMOTE_SHA" ] && exit 0`. Step 3's own health checks pass in that skewed state. Added an explicit rebuild, and said when it can be skipped. 5. docs/02-constraints/README.md -- which CLAUDE.md's critical-rules section names as the full list -- had no production-branch rule at all. This PR found and fixed exactly that gap in CONTRIBUTING.md and left the file CLAUDE.md points at. Added, including the never-force-push and must-stay-unprotected constraints. P2s: the ADR said CI "runs on main only" (it triggers on main push+PR and never on production -- same conclusion, wrong sentence); the Decision section described the OTA publish as unconditional when the whole block is gated behind FIRMWARE_AUTO_OTA=1; and chapter 11's "how to avoid" paragraph still led with the history-rewrite scenario the correction above it calls fictional -- it now leads with the real rule (no promotion mechanism, no staleness signal). Verified: bash -n and shellcheck -S info clean, the 11-case rollback harness passes, all repo gates green. Co-Authored-By: WOZCODE <contact@withwoz.com>
Implements the decision recorded in #152:
productionbecomes the release branch for both the web services and firmware OTA.mainstays the integration line; a release is a deliberate fast-forward ofproductiononto a chosenmaincommit, which the on-hostscripts/deploy.shtimer (BRANCH=production) then deploys.What ships
scripts/deploy.sh:BRANCH="main"→"production"; firmware publish +prod-*tags rideproduction.productionis the gated release branch for both tracks,mainis integration, a release is a fast-forward.docs/02-constraints/: the branch model now appears in both files CLAUDE.md names as authoritative.The four P0s review caught
1. The documented cutover would have reverted production. It said
git checkout production && git reset --hard origin/productionwith no promotion step. Butorigin/productionlagsmain:Running it today rolls the live host back past #193, #196 and #222, and leaves the checkout on
productionrunning a driver that tracksmain— which takes the old barelog "skip"; exit 0path. Deploys stop silently, permanently, and the alert that would have said so is reverted away with everything else. The procedure is now ordered promote → verify → checkout, with the verify command (git show origin/production:scripts/deploy.sh | grep '^BRANCH='must printproduction) in both the doc and the Discord alert body.2. Flipping
BRANCHsilently stops all deploys until the host is cut over.deploy.shexits early when the checkout ≠BRANCH. That was harmless whileBRANCH=main; it is not now. It sends one Discord alert naming the exact commands, with a marker so it doesn't repeat every two minutes, and clears once the host is correct.3. The false history claim survived in
firmware-release.mdafter being retracted in the other two docs — i.e. the runbook, the file read while cutting a release, was the one still carrying the fabrication. Fixed and swept.4. "verified stale" wasn't verified. The enumeration omitted four test files and never mentioned the ESP commits. Re-derived: the archived tip uses
esp_task_wdt_reconfigure, which appears nowhere onmain.mainfixes the same AP-mode reboot loop differently (≥60 sTASK_WDT_TIMEOUT_S+runAccessPoint()feeding the WDT, recorded as fixed in troubleshooting.md), so nothing live is lost — but "already exists onmain" was the wrong description.Also fixed: the OTA notification cited ADR-028 instead of ADR-030 and told the operator to
git checkout mainwithout saying not on this host (doing so trips the new guard); the cutover now rebuilds artifacts, sincegit reset --hardrestores source only and no later tick repairs it; and "a real promotion gate" is downgraded to gate-by-convention, with an explicit warning thatproductionmust stay unprotected orpublish_firmware's push is rejected and the fleet gets firmware whose bump isn't in git.The reconciliation force-reset is already done (
origin/production=a0e7374, an ancestor ofmain). What remains, after merge:Optional but recommended — the archive tag is local-only, so those 25 commits currently survive on the remote only via a stale branch
clean_gonewould delete:Follow-up
#225 —
publish_firmwarecommits toproduction, the one automated violation of the fast-forward invariant. Previously "tracked as a follow-up" with nothing behind it.Testing
No unit or e2e layer covers deploy-branch configuration. Verified:
bash -n,shellcheck -S info(allscripts/*.sh+ESP32-CAM/build.sh), an 11-case harness over the realrollback()/rollback_health_targets(), a 13-casechanged_matchgate matrix, and all five repo gate scripts. The merge withmainwas checked symbol-by-symbol to confirm #196's hardening survived intact. The live cutover is an operator step and cannot be exercised from CI.🤖 Reviewed across two rounds with the repo's
senior-reviewergate.