Skip to content

fix(platform): stop before the traffic pause when an upgrade cannot finish - #437

Merged
Brooooooklyn merged 2 commits into
mainfrom
fix/platform-upgrade-drift
Oct 16, 2026
Merged

Brooooooklyn merged 2 commits into
mainfrom
fix/platform-upgrade-drift

Conversation

@Brooooooklyn

@Brooooooklyn Brooooooklyn commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Problem

An operator upgraded a self-hosted platform from 0.27 to 1.1. Every upgrade attempt failed after Void paused application traffic, and every retry failed the same way.

void platform upgrade
  close traffic gate ──▶ application traffic paused
  upload platform Worker (cf CLI, no terminal)
     └─ Worker was changed in the dashboard / with the API
        (for example a hand-set EMAIL_FROM)
        cf strict mode: "Aborting the upload operation because of conflicts.
                         ... remove the `--strict` flag"
  ✗ run fails, gate stays closed ──▶ traffic stays paused
  retry ──▶ same abort (--yes does not answer cf's prompt)

Causes, checked against cf 1.0.0-beta.17 and live throwaway Workers:

Cause Detail
cf strict mode is hard-coded for deploy and versions upload With last_deployed_from = api, or dash with a config diff that is not add-only, cf asks. Without a TTY it aborts. No flag or env var skips it.
Void runs cf with stdio: ['ignore', 'pipe', 'pipe'] So cf can never ask.
The check happens after the traffic gate The installer closes the gate before the first Worker upload.
Operators had to edit Workers by hand The installer never set EMAIL_FROM, and the API README told operators to set it in the dashboard.
Void's own trigger-record check An API-edited Worker's active version has no Void record, so the deploy refused it ("predates safe trigger snapshots"), also after the gate.
Email permissions The runtime token needs Email Routing permissions. Nothing checked them before the pause, and the late error did not name a permission.

What this PR changes

upgrade | repair | enable | rollback | install --resume
 │
 ├─ find platform Workers the run uploads that need approval:      (no Worker change yet)
 │    last_deployed_from dash | api
 │    or active version without a Void deployment record
 │    or (interrupted cutover) the version the run restores has no record
 │    unreadable (after 2 retries) or no single version at 100% ──▶ STOP
 │
 ├─ email on? check runtime token: Zone Read, Zone Settings,         (no Worker change yet)
 │    Email Routing Rules, Email Routing Addresses (read-only GETs)
 │
 ├─ none ────────────────▶ normal run (unchanged)
 ├─ no TTY or --yes ─────▶ STOP "No platform Worker changed. This command did not pause
 │                          application traffic. ... Run this command once in a terminal"
 └─ TTY ─▶ note + "Replace the configuration of these Workers?" (default No)
             ├─ No ──────▶ STOP (same wording)
             └─ Yes
                 ▼
installPlatform
 ├─ provision resources, prepare database
 ├─ for dash/api Workers: pause spinner ─▶ cf versions upload, stdio inherited
 │    (cf shows its diff and asks; version is NOT deployed) ─▶ re-read marker
 │    └─ refused ─▶ STOP "No deployment changed. This command did not pause application traffic."
 ├─ traffic gate
 └─ deploy (agreed Workers may have an active version without a Void record)

Other changes:

  • Email preflight. When platform email is on, the runtime token's Email Routing permissions are checked before the pause. A denial names each missing permission and keeps the saved token (edit its permissions; the value stays the same). A 401 on the token saved on this machine warns and continues, because the platform may use a newer token. A late shared-email setup failure now names the likely permission.
  • Default sender. Platform mail (invitations, welcome, deploy failures, error alerts) sends from <display name> <noreply@<EMAIL_SENDER_DOMAIN>>. An explicit EMAIL_FROM still wins. Void Cloud's hosted sender does not change.
  • cf strict-abort text. It no longer says "Wrangler" or "remove the --strict flag". It names an outside change only for the marker cases. A remote secret with the same name as a Void value says which secret to delete. A workflow owned by another Worker says to delete or rename it.
  • Docs. Maintenance (upgrade, changes made outside Void, a paused-traffic note), credentials (email permissions before an upgrade), email (default sender), CI, CLI reference, and the API README no longer tells operators to set EMAIL_FROM / AUTH_REDIRECT_URI by hand.

Decisions

Topic Decision
Override without asking No new flag. --yes is not consent to replace outside changes. cf has no non-interactive override, so CI stops and names the command to run once in a terminal.
Where cf asks Inside installPlatform, after resources and the database are ready, before the gate. An upgrade that adds a resource (for example the deployment queue) needs it to exist before cf can upload a valid version.
What cf uploads An undeployed version built from the run's own config. The real deploy follows in the same run.
dash markers Always flagged, also when cf would accept an add-only diff. Void owns these Workers, so it shows the change and asks, and never replaces it without approval.
Failed marker or version read Retry twice on 429, 5xx or a network error, then stop before any Worker change. Nothing has changed at that point, so a stop costs only a retry; a guess could pause traffic and then fail. A 404 means the Worker does not exist, unless its marker shows an outside change.
Email check scope Only when platform email is on and the installation is not staying disabled. A platform without email makes no new request.
Saved token on this machine 403 stops with text that names the saved token and the env var. 401 warns and continues.
Sender override No new installer option. EMAIL_FROM set in the dashboard is a change outside Void, and the next upgrade asks before it replaces it.

Behavior changes to note in the release

  • void platform upgrade|repair|enable|rollback and install --resume stop before they change a platform Worker or pause traffic when a platform Worker was changed in the Cloudflare dashboard or API. In a terminal they ask first. In CI, run the command once in a terminal.
  • These commands, and their --plan, also stop when Cloudflare cannot answer for a platform Worker after 2 retries, or when a platform Worker has no single version at 100% of traffic.
  • With platform email on, these commands and a fresh install check the runtime token's Email Routing permissions before they pause traffic. This includes Account → Email Routing Addresses: main's docs already listed it, but a repeat upgrade did not use it, so a token without it now stops the upgrade.
  • Self-hosted platform mail sends from noreply@<mail domain> by default. EMAIL_FROM is optional. A platform that had email on but no EMAIL_FROM starts to send mail after the upgrade: invitations report "sent", and the first error alert per project can cover all errors since its alerts started to fail.

Known limits

  • cf's strict mode also asks about a remote secret with the same name as a Void value, and about a workflow that another Worker owns. These do not change last_deployed_from, so Void cannot see them before the gate. The error after the gate now says what to delete.
  • A dash Worker whose diff is add-only now stops CI, where main uploaded it without asking.
  • The email check sends read requests only. A token with Read instead of Edit access to Email Routing, or without Email Sending, passes it and fails after the pause; the error then names the permission. A safe write probe does not exist (an address POST sends mail).
  • The database lifecycle lock can add a column to its own table before the check runs. So the stop text says "No platform Worker changed", not "Nothing changed".
  • A run that reuses the active version (no upload) stops CI when the marker is dash or api, for example a repair after a dashboard rollback to Void's own version. Main did not run cf on that path, so it passed.
  • A platform release that adds a Durable Object class with a binding cannot upload to an existing platform Worker (same on main): an undeployed version cannot bind a class that Cloudflare has not created yet. No release does this yet. When a Worker was changed outside Void, the review upload then fails before the pause, so traffic is not paused. The deploy path needs its own change.
  • install --resume and --plan cannot read the database before the run. During the one-time change to configurable sign-in, they can ask once more than needed. They never stop after the pause.
  • Not checked live: a real dashboard edit (dash), and whether Void's custom-domain attach changes the marker (the test account has no zone). cf aborts on api with the same rule when Void uploads, so an api marker adds a new stop only on the reuse path above.

Testing

Check Result
vp run test (changed-file selection) 550 files passed, 1 skipped; 11371 tests passed (before the last fix; focused platform files after it: 493 passed)
Focused files (outside changes, installer, cutover, management, lifecycle, setup flow, errors, provision, operation process, api email) all pass; each new test fails without its change
vp run check, vp lint, vp fmt --check, vp run build:core pass
Live, throwaway Workers on the e2e account (all deleted) cf under a PTY: answer No keeps api; Yes uploads an undeployed version and resets the marker to cf_cli; consent then deploy succeeds; API settings PATCH removes the Void record; Void's own operations keep cf_cli
Live, read-only email probes all four GETs answer 200 on a mail zone; unknown zone and invalid token are reported as expected
Real cf 1.0.0-beta.17 against a mock API marker, secret and workflow aborts give the new messages
PTY with expect Ctrl-C at the cf prompt ends the run with no gate write and no leftover child

🤖 Generated with Claude Code

https://claude.ai/code/session_0123r5HnanmW2VUrZ7VWDufH

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T16:39:09.511292Z 87e43b4 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9a224ec4e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/void/src/cli/platform-installer-workers.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bbe98e36c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/void/src/cli/platform-installer-outside-changes.ts Outdated
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://fix-platform-upgrade-drift-void-cloud.voidzero-docs.workers.dev (commit 87e43b4)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://575ed536-void-cloud.voidzero-docs.workers.dev 87e43b4 2026-10-15T16:44:44.405Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://3bfde439-void-cloud.voidzero-docs.workers.dev f923fd9 2026-10-15T16:24:42.405Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://8546cd0d-void-cloud.voidzero-docs.workers.dev bbe98e3 2026-10-15T15:51:13.540Z Visit the dashboard ↗

Brooooooklyn and others added 2 commits October 16, 2026 00:33
…inish

A platform Worker changed in the Cloudflare dashboard or API made every
upgrade fail after traffic was paused: the Cloudflare CLI asks before it
replaces such a change, and Void runs it without a terminal. Void now
finds these Workers before it changes anything. In a terminal it asks,
then lets the Cloudflare CLI show the changes and upload an undeployed
version after resources are ready and before traffic pauses. Without a
terminal it stops and names the command to run once in a terminal.

The runtime token's Email Routing permissions are checked before the
pause when platform email is on, and the late setup error names the
missing permission. Platform mail now has a default sender at
noreply@<mail domain>, so operators no longer set EMAIL_FROM by hand.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0123r5HnanmW2VUrZ7VWDufH
…annot be read

A failed Cloudflare read counted as "no change", so the run could pause
traffic and then fail in the Cloudflare CLI or in Void's version-record
check. Void now retries a read that fails with 429, 5xx or a network
error, then stops before it changes a platform Worker. A Worker without
one version at 100% of traffic stops the run too, because the deploy
needs one. The stop messages now say what is true at that point: no
platform Worker changed and the command did not pause traffic.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0123r5HnanmW2VUrZ7VWDufH
@Brooooooklyn
Brooooooklyn force-pushed the fix/platform-upgrade-drift branch from f923fd9 to 87e43b4 Compare October 15, 2026 16:33
@Brooooooklyn
Brooooooklyn merged commit 5802901 into main Oct 16, 2026
20 checks passed
@Brooooooklyn
Brooooooklyn deleted the fix/platform-upgrade-drift branch October 16, 2026 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant