Skip to content

fix(release): survive a transient Apple notary fault instead of losing the release - #927

Open
lacymorrow wants to merge 2 commits into
mainfrom
LAC-6331-juno-release-build-v0-8-253-failed-nothing-published-gh-926
Open

lacymorrow wants to merge 2 commits into
mainfrom
LAC-6331-juno-release-build-v0-8-253-failed-nothing-published-gh-926

Conversation

@lacymorrow

Copy link
Copy Markdown
Owner

Paperclip: LAC-6331 · GitHub: #926

What happened

The release build for v0.8.253 (run 41191714061) failed at notarization, so nothing was published. It is not an isolated failure: six release builds between 2026-10-27 and 2026-10-31 died at a notary call. v0.8.251 and v0.8.253 both have tags and no release. Everyone on the prerelease channel simply stopped receiving builds.

Run Failed on
10-30 04:30 HTTP status code: 401. Invalid credentials
10-30 13:50 HTTP status code: 401
10-30 21:53 getaddrinfo ENOTFOUND api.github.com
10-29 23:13 / 22:17 / 15:31 HTTP status code: 401
10-29 06:24 NSURLErrorDomain Code=-1009 "offline" against appstoreconnect.apple.com/notary/v2/asp
10-27 15:01 empty notary error
10-31 10:11 (this one) HTTP status code: 401

The 401 is not a bad password, and the ticket's premise was wrong

LAC-6331 said to hand APPLE_PASSWORD rotation to Lacy if the failure was a 401. That had already been done and did not work. APPLE_PASSWORD was replaced at 2026-10-31T08:20Z; the run two hours later 401'd on the new value.

When (UTC) APPLE_PASSWORD Result
10-30 04:30 old (set 09-12) 401
10-30 13:50 old 401
10-30 14:18 old ✅ success
10-31 00:46 old ✅ success
10-31 08:20 rotated —
10-31 10:11 new 401

A credential that succeeds twice between 401s is valid. The mechanism is in the 10-29 failure, which names the endpoint: notary/v2/asp is the app-specific-password token exchange, so a runner that cannot reach Apple gets 401 Invalid credentials rather than a network error. Today's notarization failed 3.6 seconds after starting (11:04:36.9 → 11:04:40.5), at the auth exchange, before any upload — after a 50-minute compile.

So no secret rotation is needed and nothing is handed to Lacy. Flagging for Security as the issue asked, since the diagnosis concerns a signing secret: APPLE_PASSWORD is not compromised or expired, and no secret changes in this PR.

Changes

All in .github/workflows/release-tauri.yml.

  1. Check the notary service, before the build. notarytool history is the smallest authenticated call there is, retried 5 times over ~3 minutes. A sustained outage now fails red in minutes instead of 50; a brief blip is absorbed. It reports 401 and unreachable distinctly, because that distinction is what decides a human's next move, and the raw error text has already misled once.
  2. One retry of the Tauri build. The preflight cannot catch an outage that begins after the build starts, which is exactly today's failure. ./target is already populated in the same job, so the retry skips compilation and goes straight back to bundling — which also means a genuine compile error fails the retry in minutes rather than repeating the full build. GitHub Actions has neither step retry nor YAML anchors, so the block is a hand copy and carries a lockstep comment.
  3. The report job no longer asserts a cause it cannot observe. It hardcoded "Notarization failures (HTTP 401) mean the APPLE_PASSWORD app-specific password needs replacing." That job opened Release builds are failing #926, so the ticket arrived carrying a diagnosis generated by a string constant rather than by reading the log — and a password was rotated on the strength of it. It now names the step that actually failed (from the jobs API, which answers immediately; run logs are usually still locked that soon) and states the evidence. Rewritten as a heredoc so the failure path carries no shell metacharacters; the old quoted form also indented every paragraph by 10 spaces, which GitHub rendered as code blocks.

Testing

No Rust or frontend code changed; this is workflow-only.

  • actionlint clean; YAML parses and step order is correct.
  • shellcheck clean on both new run blocks (shellcheck is installed locally, so actionlint's integration was live).
  • Both scripts executed against stubbed xcrun/gh, covering every branch:
    • preflight: credentials accepted on attempt 1 → exit 0; fails twice then succeeds → exit 0; persistent 401 → exit 1 with the 401-specific message; unreachable → exit 1 with the network message.
    • report: rendered the real issue body for a simulated failure, confirming variable expansion, the failed-step name, and correct markdown.

Recovery of v0.8.253

Re-dispatched as run 41210696122 (--ref main -f tag=v0.8.253 -f prerelease=true, matching how release-every-merge.yml dispatches). Note that workflow_dispatch takes the workflow file from the dispatched ref while checking out inputs.tag, so once this merges, a re-dispatch of any stranded tag gets the retry.

Verification per the issue ("a published release containing the version") is tracked on LAC-6331 and not claimed here: this PR is the durable fix, the re-run is the recovery.

Follow-up filed

The structural fix is to stop using app-specific-password auth at all — App Store Connect API key auth (APPLE_API_KEY / APPLE_API_ISSUER / APPLE_API_KEY_PATH) signs a JWT locally and never touches /asp. That needs a new .p8 minted in App Store Connect, which is owner-only, so it is filed separately rather than smuggled in here.

@lacymorrow

Copy link
Copy Markdown
Owner Author

Verified the three assumptions this PR's retry rests on, by reading tauri-apps/tauri-action@v0's source rather than assuming:

  1. The retry really is cheap. src/build.ts has no clean/rm/rimraf of the target dir, so the second attempt reuses the ./target the first one populated in the same job and skips compilation. (Grep hits for "clean" in that file are all targetInfo.platform === 'macos' comparisons.)
  2. A second attempt does not duplicate the release. src/create-release.ts exports getOrCreateRelease, which calls getReleaseByTag first and updateRelease when one exists.
  3. A failed attempt leaves nothing behind. In src/index.ts, buildProject() runs at line 39 and getOrCreateRelease only at line 117, so a build that dies at notarization never creates a release. That matches what we observed: v0.8.253 has a tag and no release.

Worth noting for the reviewer: index.ts already wraps getOrCreateRelease in retry(...), and uploadReleaseAssets takes a retryAttempts argument. The action's authors had already concluded that retry is the right answer for transient faults against a remote service — they just never covered the build/notarize path, which is the expensive one and the one that has been failing. This PR closes that gap rather than introducing a new policy.

lacymorrow and others added 2 commits October 31, 2026 17:17
…g the release

Six release builds between 2026-10-27 and 2026-10-31 died at a notary call
and published nothing, so v0.8.251 and v0.8.253 have tags and no release.
Four failed on "HTTP status code: 401. Invalid credentials", one on
NSURLErrorDomain -1009 "the Internet connection appears to be offline"
against appstoreconnect.apple.com/notary/v2/asp, one on an empty notary
error. Each burned ~50 minutes of compilation first; today's notarization
failed 3.6s after starting, at the auth exchange, before any upload.

The 401 is not a verdict on the password. notary/v2/asp IS the
app-specific-password token exchange, so a runner that cannot reach it gets
a 401 rather than a network error. The same unchanged credential succeeded
at 2026-10-30T14:18Z and 2026-10-31T00:46Z in between two 401s, and the
replacement set at 2026-10-31T08:20Z then 401'd at 10:11Z.

- Check the notary service before the build. notarytool history is the
  smallest authenticated call there is, retried 5 times over ~3 minutes. A
  sustained outage now fails red in minutes instead of 50, and a brief blip
  is absorbed. It reports 401 and unreachable distinctly, because that
  distinction decides what a human should do next.
- Retry the Tauri build once, for an outage that starts after the build
  does, which is what happened today. ./target is already populated in the
  same job, so the retry skips compilation; a genuine compile error also
  fails in minutes rather than repeating the full build.
- Stop the report job asserting a cause it cannot observe. It hardcoded
  "Notarization failures (HTTP 401) mean the APPLE_PASSWORD app-specific
  password needs replacing". That sentence opened #926, a password was
  rotated on the strength of it, and the next build 401'd on the new one. It
  now names the step that actually failed, from the jobs API, and states the
  evidence. Written as a heredoc so the failure path carries no shell
  metacharacters; the old quoted form also indented every paragraph by 10
  spaces, which GitHub rendered as code blocks.

Verified: actionlint clean, YAML parses, shellcheck clean on both new run
blocks, and both scripts executed against stubbed xcrun/gh covering accept,
transient-recovery, persistent-401 and unreachable.

Refs LAC-6331, #926
…SWORD is even shaped like one

Three consecutive dispatched release builds have now failed on notarization
401 over eight hours (10:11Z, 14:33Z, 18:12Z), all of them after the
APPLE_PASSWORD rotation at 08:20Z, with no success since. Before that
rotation the old value succeeded roughly half the time. That is a regime
change correlated with the rotation, not the intermittency the first commit
addressed, and it points at the stored value itself.

Nothing in the pipeline could tell the difference, because all three causes
print the identical 401: a revoked password, an unreachable notary, and a
password with an invisible extra character. So inspect the credential's
shape, which nobody had done.

- Trim stray whitespace off the single-line credentials before exporting
  them. A secret pasted into GitHub's web form easily carries a trailing
  newline, and `echo "${!var}"` into the heredoc preserves it, so the notary
  sees a password one character longer than the one Apple issued.
  APPLE_CERTIFICATE is excluded on purpose: it is multi-line base64 and its
  internal newlines are load-bearing. Warns per variable when it strips
  anything, naming which one.
- Report whether APPLE_PASSWORD matches Apple's app-specific-password shape
  (exactly xxxx-xxxx-xxxx-xxxx). A regular Apple ID password in that secret
  returns 401 forever and no amount of retrying helps. Only the length and a
  yes/no are printed, never any part of the value.

Verified by executing the extracted step against six credential states:
correct password, trailing newline, trailing space, a trailing newline on
APPLE_TEAM_ID instead, and a non-app-specific password. Each produced the
right warning and the stored bytes came out clean (checked with od -c).
actionlint and shellcheck clean; bash 3.2 compatible, which the runner has.

Refs LAC-6331, #926

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@lacymorrow
lacymorrow force-pushed the LAC-6331-juno-release-build-v0-8-253-failed-nothing-published-gh-926 branch from 92bce3b to e47837c Compare October 31, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant