Repository navigation
fix(release): survive a transient Apple notary fault instead of losing the release - #927
Open
lacymorrow wants to merge 2 commits into
Open
lacymorrow wants to merge 2 commits into
lacymorrow wants to merge 2 commits into
Conversation
Owner
Author
|
Verified the three assumptions this PR's retry rests on, by reading
Worth noting for the reviewer: |
…g the release Six release builds between 2026-10-27 and 2026-10-31 died at a notary call and published nothing, so v0.8.251 and v0.8.253 have tags and no release. Four failed on "HTTP status code: 401. Invalid credentials", one on NSURLErrorDomain -1009 "the Internet connection appears to be offline" against appstoreconnect.apple.com/notary/v2/asp, one on an empty notary error. Each burned ~50 minutes of compilation first; today's notarization failed 3.6s after starting, at the auth exchange, before any upload. The 401 is not a verdict on the password. notary/v2/asp IS the app-specific-password token exchange, so a runner that cannot reach it gets a 401 rather than a network error. The same unchanged credential succeeded at 2026-10-30T14:18Z and 2026-10-31T00:46Z in between two 401s, and the replacement set at 2026-10-31T08:20Z then 401'd at 10:11Z. - Check the notary service before the build. notarytool history is the smallest authenticated call there is, retried 5 times over ~3 minutes. A sustained outage now fails red in minutes instead of 50, and a brief blip is absorbed. It reports 401 and unreachable distinctly, because that distinction decides what a human should do next. - Retry the Tauri build once, for an outage that starts after the build does, which is what happened today. ./target is already populated in the same job, so the retry skips compilation; a genuine compile error also fails in minutes rather than repeating the full build. - Stop the report job asserting a cause it cannot observe. It hardcoded "Notarization failures (HTTP 401) mean the APPLE_PASSWORD app-specific password needs replacing". That sentence opened #926, a password was rotated on the strength of it, and the next build 401'd on the new one. It now names the step that actually failed, from the jobs API, and states the evidence. Written as a heredoc so the failure path carries no shell metacharacters; the old quoted form also indented every paragraph by 10 spaces, which GitHub rendered as code blocks. Verified: actionlint clean, YAML parses, shellcheck clean on both new run blocks, and both scripts executed against stubbed xcrun/gh covering accept, transient-recovery, persistent-401 and unreachable. Refs LAC-6331, #926
…SWORD is even shaped like one
Three consecutive dispatched release builds have now failed on notarization
401 over eight hours (10:11Z, 14:33Z, 18:12Z), all of them after the
APPLE_PASSWORD rotation at 08:20Z, with no success since. Before that
rotation the old value succeeded roughly half the time. That is a regime
change correlated with the rotation, not the intermittency the first commit
addressed, and it points at the stored value itself.
Nothing in the pipeline could tell the difference, because all three causes
print the identical 401: a revoked password, an unreachable notary, and a
password with an invisible extra character. So inspect the credential's
shape, which nobody had done.
- Trim stray whitespace off the single-line credentials before exporting
them. A secret pasted into GitHub's web form easily carries a trailing
newline, and `echo "${!var}"` into the heredoc preserves it, so the notary
sees a password one character longer than the one Apple issued.
APPLE_CERTIFICATE is excluded on purpose: it is multi-line base64 and its
internal newlines are load-bearing. Warns per variable when it strips
anything, naming which one.
- Report whether APPLE_PASSWORD matches Apple's app-specific-password shape
(exactly xxxx-xxxx-xxxx-xxxx). A regular Apple ID password in that secret
returns 401 forever and no amount of retrying helps. Only the length and a
yes/no are printed, never any part of the value.
Verified by executing the extracted step against six credential states:
correct password, trailing newline, trailing space, a trailing newline on
APPLE_TEAM_ID instead, and a non-app-specific password. Each produced the
right warning and the stored bytes came out clean (checked with od -c).
actionlint and shellcheck clean; bash 3.2 compatible, which the runner has.
Refs LAC-6331, #926
Co-Authored-By: Paperclip <noreply@paperclip.ing>
lacymorrow
force-pushed
the
LAC-6331-juno-release-build-v0-8-253-failed-nothing-published-gh-926
branch
from
October 31, 2026 21:18
92bce3b to
e47837c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paperclip: LAC-6331 · GitHub: #926
What happened
The release build for v0.8.253 (run 41191714061) failed at notarization, so nothing was published. It is not an isolated failure: six release builds between 2026-10-27 and 2026-10-31 died at a notary call. v0.8.251 and v0.8.253 both have tags and no release. Everyone on the prerelease channel simply stopped receiving builds.
HTTP status code: 401. Invalid credentialsHTTP status code: 401getaddrinfo ENOTFOUND api.github.comHTTP status code: 401NSURLErrorDomain Code=-1009 "offline"againstappstoreconnect.apple.com/notary/v2/aspHTTP status code: 401The 401 is not a bad password, and the ticket's premise was wrong
LAC-6331 said to hand
APPLE_PASSWORDrotation to Lacy if the failure was a 401. That had already been done and did not work.APPLE_PASSWORDwas replaced at 2026-10-31T08:20Z; the run two hours later 401'd on the new value.APPLE_PASSWORDA credential that succeeds twice between 401s is valid. The mechanism is in the 10-29 failure, which names the endpoint:
notary/v2/aspis the app-specific-password token exchange, so a runner that cannot reach Apple gets401 Invalid credentialsrather than a network error. Today's notarization failed 3.6 seconds after starting (11:04:36.9 → 11:04:40.5), at the auth exchange, before any upload — after a 50-minute compile.So no secret rotation is needed and nothing is handed to Lacy. Flagging for Security as the issue asked, since the diagnosis concerns a signing secret:
APPLE_PASSWORDis not compromised or expired, and no secret changes in this PR.Changes
All in
.github/workflows/release-tauri.yml.Check the notary service, before the build.notarytool historyis the smallest authenticated call there is, retried 5 times over ~3 minutes. A sustained outage now fails red in minutes instead of 50; a brief blip is absorbed. It reports 401 and unreachable distinctly, because that distinction is what decides a human's next move, and the raw error text has already misled once../targetis already populated in the same job, so the retry skips compilation and goes straight back to bundling — which also means a genuine compile error fails the retry in minutes rather than repeating the full build. GitHub Actions has neither step retry nor YAML anchors, so the block is a hand copy and carries a lockstep comment.reportjob no longer asserts a cause it cannot observe. It hardcoded "Notarization failures (HTTP 401) mean theAPPLE_PASSWORDapp-specific password needs replacing." That job opened Release builds are failing #926, so the ticket arrived carrying a diagnosis generated by a string constant rather than by reading the log — and a password was rotated on the strength of it. It now names the step that actually failed (from the jobs API, which answers immediately; run logs are usually still locked that soon) and states the evidence. Rewritten as a heredoc so the failure path carries no shell metacharacters; the old quoted form also indented every paragraph by 10 spaces, which GitHub rendered as code blocks.Testing
No Rust or frontend code changed; this is workflow-only.
actionlintclean; YAML parses and step order is correct.shellcheckclean on both newrunblocks (shellcheckis installed locally, so actionlint's integration was live).xcrun/gh, covering every branch:Recovery of v0.8.253
Re-dispatched as run 41210696122 (
--ref main -f tag=v0.8.253 -f prerelease=true, matching howrelease-every-merge.ymldispatches). Note thatworkflow_dispatchtakes the workflow file from the dispatched ref while checking outinputs.tag, so once this merges, a re-dispatch of any stranded tag gets the retry.Verification per the issue ("a published release containing the version") is tracked on LAC-6331 and not claimed here: this PR is the durable fix, the re-run is the recovery.
Follow-up filed
The structural fix is to stop using app-specific-password auth at all — App Store Connect API key auth (
APPLE_API_KEY/APPLE_API_ISSUER/APPLE_API_KEY_PATH) signs a JWT locally and never touches/asp. That needs a new.p8minted in App Store Connect, which is owner-only, so it is filed separately rather than smuggled in here.