Skip to content

Mirror native artifacts to S3 and check all destinations independently - #318

Merged
avallete merged 4 commits into
mainfrom
claude/mirroring-resilience-s3-fallback-8nm0qz
Sep 23, 2026
Merged

Mirror native artifacts to S3 and check all destinations independently#318
avallete merged 4 commits into
mainfrom
claude/mirroring-resilience-s3-fallback-8nm0qz

Conversation

@avallete

@avallete avallete commented Sep 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Problem

With ECR Public not ready yet (missing cli/* repositories and permissions), every mirror path failed as a whole:

  • ecr-mirror.ts sync --request stopped at the first image that didn't show up on ECR, after its 15-minute poll.
  • Nothing ever checked the S3 copy of the natives, so it could be neither confirmed nor repaired.
  • The audit covered the whole release history, which isn't needed.

Behavior

  • Each destination is checked and reported on its own: the image on ECR Public, the native tags on ECR Public, and the natives on S3. An S3 copy counts as in sync when the public SHA256SUMS object hashes to the checksum layer of the native on GHCR.
  • sync --request sends every out-of-sync release first, then waits for all of them against one shared timeout. An unreachable ECR costs a single timeout, and the S3 copy of every release still lands.
  • By default the audit and the backfill only cover the latest release of each service release line; postgres keeps one per major. History is audited only with --all / all_releases: true.
  • Drift on the ECR image or on S3 fails the audit. Drift on ECR native tags only fails it under ECR_MIRROR_REQUIRE_NATIVES=1, as before.
  • At release time, mirrored (the ECR lines in the release notes) depends only on the image check. Mirroring still never blocks the GitHub release.

Targeting releases

The services input of ECR mirror check (and sync locally) accepts services, which means their latest release, or exact releases written SERVICE:VERSION, space-separated:

gh workflow run ecr-mirror-check.yml -f request=true                        # latest of every service
gh workflow run ecr-mirror-check.yml -f request=true -f services=postgrest:v16.2
gh workflow run ecr-mirror-check.yml -f request=true \
  -f services="postgres:15.14.1.175 postgres:17.6.1.175 realtime"
bun scripts/ecr-mirror.ts sync --request postgrest:v16.2                    # locally

Natives are mirrored from the GHCR <version>-native-<target> tags. A release published before those tags existed has nothing to copy until it is rebuilt with service-release.yml and force=true, which republishes its natives and dispatches the mirror. Today that applies to every latest release except mailpit v1.31.2.

This is documented in the README under "Backfilling mirrors" and in docs/design/ecr-mirror-dispatch.md.

Rollout

Merge supabase/cli#6716 first. Dispatches run the mirror workflow from cli's develop, and that PR stops the S3 upload from waiting on the ECR copy.

Verification

  • bun test ./scripts/ecr-mirror.test.ts ./scripts/publish-native-oci.test.ts passes. New cases cover: ECR never converging while S3 lands for every release, S3 drift while ECR is in sync, the image and S3 reported separately at release time, latest per release line versus --all, and service and SERVICE:VERSION filters.
  • Against the live release list, the default selection is the 14 latest releases (two for postgres).
  • The bun build check and scripts/test-external-workflows.sh pass.
  • Not run end to end against GitHub, ECR or S3.

🤖 Generated with Claude Code

https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

@claude
ci: check and backfill ECR Public and S3 mirrors independently
ce1d32b 
An ECR Public failure (missing cli/* repositories or permissions) made
every mirror path fail as a whole: the backfill aborted at the first image
that did not verify after its 15-minute poll, and S3 natives were never
audited, so nothing could confirm or repair the S3 copy.

- `request` and `sync --request` now wait for the ECR image, ECR natives
  and S3 natives against one shared deadline and report each destination
  separately. `sync --request` dispatches every out-of-sync release before
  waiting, so an unreachable destination costs one timeout and the S3 copy
  of every release still lands.
- S3 natives are verified by hashing the anonymous SHA256SUMS object
  against the GHCR native's checksum layer; S3 drift fails the audit.
- `sync` takes optional service names, exposed as the `services` input of
  the check workflow, to backfill in slices.
- The release-time mirror job records `mirrored` from the image check
  alone, so an S3 failure does not drop the ECR lines from release notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p
@claude
ci: allow the mirror check to target single releases
627c1f8 
`sync` and the check workflow's `services` input now also accept
`SERVICE:VERSION`, so one release can be audited or backfilled without
dispatching every version of its service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p
@avallete avallete mentioned this pull request Sep 23, 2026
Merged
@claude
docs: explain how to backfill mirrors for one release
94e806c 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p
@claude
ci: audit only the latest release of each line by default
7c4038a 
The mirror check and its backfill now cover the newest release of each
service release line (postgres keeps one per major); history is audited
only with `--all` / `all_releases: true`. An explicit `SERVICE:VERSION`
still selects that exact release, so an older version rebuilt with
`force=true` can be mirrored on demand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p
@avallete
avallete merged commit bb121ad into main Sep 23, 2026
2 checks passed
@avallete
avallete deleted the claude/mirroring-resilience-s3-fallback-8nm0qz branch September 23, 2026 09:23
pull Bot pushed a commit to oogalieboogalie/cli that referenced this pull request Sep 24, 2026
@avallete @claude
feat(stack): fall back to the S3 and ECR Public slim mirrors (supabas…
1725770 
…e#6716)

## Summary

Slim native artifacts and images now fall back to mirrors. The primary
source is always tried first. When every candidate fails, the user still
sees the primary's original error. A fallback failure, including a
checksum mismatch, is logged as a warning, and a fallback that succeeds
is logged.

- **Natives, downloads:** archives and manifests are tried from
`SLIM_NATIVE_GITHUB_RELEASES` first, then
`SLIM_NATIVE_SUPABASE_S3_MIRROR` (the public `supabase-cli-artifacts`
bucket).
- **Natives, checksums:** checksum sources are listed separately from
download mirrors. The checksum comes from the GitHub Release
`SHA256SUMS`, or else from the archive layer digest of
`ghcr.io/supabase/cli/<service>:<version>-native-<target>`, read with an
anonymous pull token. The bucket is never a checksum source, and an
archive from any host is only accepted if it hashes to that checksum.
- **Images:** the container runtime pulls from
`SLIM_IMAGE_GHCR_REGISTRY` first. If that fails, it pulls the same tag
and digest from `SLIM_IMAGE_SUPABASE_ECR_MIRROR`, and launches then use
the reference that was pulled.
- **More mirrors later:** checksum sources, native mirrors and image
mirrors (`slimImageMirrors`) are all ordered lists, so adding another
host is one more entry.
- **Mirror workflow:** `upload-natives-s3` in `mirror-slim-image.yml` no
longer waits for the ECR `mirror` job, so an ECR Public failure no
longer keeps natives out of S3. The job now checks the source image
digest itself before uploading.
- **Native Postgres as root:** startup fails with `PostgreSQL cannot be
run as root` before any artifact download or process launch. Docker and
Podman are unchanged.
- **Fresh init config:** an unset `env(NAME)` literal is ignored only
for `experimental.s3_*`. Other unresolved values, including auth keys,
still fail the experimental stack.
- **A service that exits before it is ready** keeps that service's
error, including a native Postgres stderr tail.

Catalog pins that have no public mirror still fail on the primary
download. Mirroring those pins is a separate release step.

ADR 0026 is updated: the earlier decision against a public S3 bucket is
superseded for native archives. S3 is not a checksum authority.

## Linked issue

None; maintainer change. The sender side is supabase/slim-services#318.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016ctyhVP3n3xNy7YRtGYp4p

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jgoux jgoux jgoux approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@avallete @jgoux @claude