Skip to content

ci(repo): mirror slim images to the central registry's ECR Public #6715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
avallete wants to merge 1 commit into from
Closed

ci(repo): mirror slim images to the central registry's ECR Public #6715

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 14 additions & 37 deletions .github/scripts/mirror-slim-image.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { describe, expect, test } from "bun:test";
import {
copyImage,
copyNatives,
ensureEcrPublicRepo,
checkEcrPublicRepo,
fetchNatives,
main,
nativesFromEvent,
Expand Down Expand Up @@ -46,7 +46,7 @@ describe("validateMirrorDispatch", () => {
version: "v16.2",
digest: DIGEST,
source: "ghcr.io/supabase/cli/postgrest:v16.2",
destination: "public.ecr.aws/supabase/cli/postgrest:v16.2",
destination: "public.ecr.aws/supabase-registry/cli/postgrest:v16.2",
});
});

Expand All @@ -58,7 +58,7 @@ describe("validateMirrorDispatch", () => {
version: "v16.2",
digest: DIGEST,
payloadSource: "ghcr.io/evil/cli/postgrest:v16.2",
payloadDestination: "public.ecr.aws/supabase/cli/postgrest:v16.2",
payloadDestination: "public.ecr.aws/supabase-registry/cli/postgrest:v16.2",
}),
).toThrow(InvalidPayloadError);
});
Expand Down Expand Up @@ -122,46 +122,23 @@ describe("verifyDigest", () => {
});
});

describe("ensureEcrPublicRepo", () => {
test("no-ops when the repository exists", async () => {
describe("checkEcrPublicRepo", () => {
test("passes when the repository exists", async () => {
const run: RunCommand = async () => ok();
const logs: string[] = [];
await ensureEcrPublicRepo({
await checkEcrPublicRepo({
service: "postgrest",
run,
log: (message) => logs.push(message),
});
expect(logs).toEqual(["ECR Public repository cli/postgrest exists"]);
});

test("creates a missing repository", async () => {
const run: RunCommand = async (argv) =>
argv.includes("describe-repositories") ? fail("not found") : ok();
const logs: string[] = [];
await ensureEcrPublicRepo({
service: "postgrest",
run,
log: (message) => logs.push(message),
});
expect(logs).toEqual(["created ECR Public repository cli/postgrest"]);
});

test("treats a create race as success", async () => {
const run: RunCommand = async () => fail("RepositoryAlreadyExistsException");
const logs: string[] = [];
await ensureEcrPublicRepo({
service: "auth",
run,
log: (message) => logs.push(message),
});
expect(logs).toEqual(["ECR Public repository cli/auth was created concurrently"]);
});

test("fails when create is denied", async () => {
const run: RunCommand = async () => fail("AccessDenied");
test("fails with the enrollment to add when the repository is missing", async () => {
const run: RunCommand = async () => fail("RepositoryNotFoundException");
await expect(
ensureEcrPublicRepo({ service: "postgrest", run, log: () => undefined }),
).rejects.toThrow(/CreateRepository/);
checkEcrPublicRepo({ service: "postgrest", run, log: () => undefined }),
).rejects.toThrow(/pulumi\/registry\/public\/cli\/cli\.yaml/);
});
});

Expand All @@ -174,7 +151,7 @@ describe("copyImage", () => {
};
await copyImage({
source: "ghcr.io/supabase/cli/postgrest:v16.2",
destination: "public.ecr.aws/supabase/cli/postgrest:v16.2",
destination: "public.ecr.aws/supabase-registry/cli/postgrest:v16.2",
digest: DIGEST,
run,
});
Expand All @@ -186,7 +163,7 @@ describe("copyImage", () => {
"--referrers",
"--digest-tags",
digestReference("ghcr.io/supabase/cli/postgrest:v16.2", DIGEST),
"public.ecr.aws/supabase/cli/postgrest:v16.2",
"public.ecr.aws/supabase-registry/cli/postgrest:v16.2",
],
]);
});
Expand Down Expand Up @@ -221,7 +198,7 @@ describe("copyNatives", () => {
"image",
"copy",
`ghcr.io/supabase/cli/postgrest@${DIGEST}`,
"public.ecr.aws/supabase/cli/postgrest:v16.2-native-linux-arm64",
"public.ecr.aws/supabase-registry/cli/postgrest:v16.2-native-linux-arm64",
],
["regctl", "manifest", "head", "ghcr.io/supabase/cli/postgrest:v16.2-native-linux-amd64"],
]);
Expand Down Expand Up @@ -263,7 +240,7 @@ describe("main", () => {
},
});
expect(code).toBe(0);
expect(fields["destination"]).toBe("public.ecr.aws/supabase/cli/postgrest:v16.2");
expect(fields["destination"]).toBe("public.ecr.aws/supabase-registry/cli/postgrest:v16.2");
});

test("copy-natives no-ops without a dispatch payload", async () => {
Expand Down
33 changes: 8 additions & 25 deletions .github/scripts/mirror-slim-image.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
*
* bun .github/scripts/mirror-slim-image.ts validate
* bun .github/scripts/mirror-slim-image.ts verify-digest
* bun .github/scripts/mirror-slim-image.ts ensure-repo
* bun .github/scripts/mirror-slim-image.ts check-repo
* bun .github/scripts/mirror-slim-image.ts copy-image
* bun .github/scripts/mirror-slim-image.ts copy-natives
* bun .github/scripts/mirror-slim-image.ts fetch-natives
Expand Down Expand Up @@ -64,7 +64,7 @@ export type MirrorIo = {
const usage = `Usage:
bun .github/scripts/mirror-slim-image.ts validate
bun .github/scripts/mirror-slim-image.ts verify-digest
bun .github/scripts/mirror-slim-image.ts ensure-repo
bun .github/scripts/mirror-slim-image.ts check-repo
bun .github/scripts/mirror-slim-image.ts copy-image
bun .github/scripts/mirror-slim-image.ts copy-natives
bun .github/scripts/mirror-slim-image.ts fetch-natives
Expand Down Expand Up @@ -114,7 +114,7 @@ export const verifyDigest = async (options: {
log(`${options.reference} resolves to ${options.digest}`);
};

export const ensureEcrPublicRepo = async (options: {
export const checkEcrPublicRepo = async (options: {
readonly service: string;
readonly run: RunCommand;
readonly log?: (message: string) => void;
Expand All @@ -134,27 +134,10 @@ export const ensureEcrPublicRepo = async (options: {
log(`ECR Public repository ${repoName} exists`);
return;
}
const created = await options.run([
"aws",
"ecr-public",
"create-repository",
"--repository-name",
repoName,
"--region",
"us-east-1",
]);
if (created.ok) {
log(`created ECR Public repository ${repoName}`);
return;
}
const detail = `${created.stdout}\n${created.stderr}`;
if (detail.includes("RepositoryAlreadyExistsException")) {
log(`ECR Public repository ${repoName} was created concurrently`);
return;
}
if (detail.trim() !== "") log(detail.trim());
const detail = `${described.stdout}\n${described.stderr}`.trim();
if (detail !== "") log(detail);
throw new InvalidPayloadError(
`ECR Public repository '${repoName}' does not exist and this role cannot create it (missing ecr-public:CreateRepository). Create it once manually — aws ecr-public create-repository --repository-name '${repoName}' --region us-east-1 — then re-run this workflow.`,
`ECR Public repository '${repoName}' is not available to this role. Add '${options.service}' to pulumi/registry/public/cli/cli.yaml in supabase/platform, then re-run this workflow once it deploys.`,
);
};

Expand Down Expand Up @@ -483,8 +466,8 @@ export const main = async (argv: ReadonlyArray<string>, io: MirrorIo): Promise<n
});
return 0;
}
if (command === "ensure-repo") {
await ensureEcrPublicRepo({
if (command === "check-repo") {
await checkEcrPublicRepo({
service: requireEnv(io.env, "SERVICE"),
run: io.run,
log,
Expand Down
2 changes: 1 addition & 1 deletion .github/scripts/slim-mirror-payload.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ const VERSION_PATTERN = /^[A-Za-z0-9._-]+$/;
const DIGEST_PATTERN = /^sha256:[0-9a-f]{64}$/;

export const SOURCE_REGISTRY = "ghcr.io/supabase/cli";
export const DEST_REGISTRY = "public.ecr.aws/supabase/cli";
export const DEST_REGISTRY = "public.ecr.aws/supabase-registry/cli";
const NATIVE_TARGETS = ["linux-arm64", "linux-amd64", "darwin-arm64"] as const;

export class InvalidPayloadError extends Error {}
Expand Down
16 changes: 9 additions & 7 deletions .github/workflows/mirror-slim-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
name: Mirror Slim Image

# Mirrors slim service images published by supabase/slim-services from ghcr.io/supabase/cli to
# public.ecr.aws/supabase/cli. The sender polls the ECR Public destination and fails its release
# unless it resolves to the exact index digest published, so the copy must be digest-preserving
# via `regctl image copy` (byte-for-byte); do not switch to `docker buildx imagetools create`,
# which can rewrite the index and change its digest.
# public.ecr.aws/supabase-registry/cli, whose repositories and push role are enrolled in
# pulumi/registry/public/cli/cli.yaml in supabase/platform. The sender polls the ECR Public
# destination and fails its release unless it resolves to the exact index digest published, so
# the copy must be digest-preserving via `regctl image copy` (byte-for-byte); do not switch to
# `docker buildx imagetools create`, which can rewrite the index and change its digest.
#
# The payload arrives with whatever authority holds the dispatch token, so it is treated as
# untrusted: names are pattern-checked, source/destination are derived in
Expand Down Expand Up @@ -104,18 +105,19 @@ jobs:
- name: Configure aws credentials
uses: aws-actions/configure-aws-credentials@cbe3b392738ccf3f987d68400dafcf4b0624a56c # v6.2.4
with:
role-to-assume: ${{ secrets.PROD_AWS_ROLE }}
role-to-assume: arn:aws:iam::945488516053:role/ecr-push-public-cli
role-session-name: mirror-slim-image-${{ github.run_id }}
aws-region: us-east-1

- name: Log in to ECR Public
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: public.ecr.aws

- name: Ensure ECR Public repository exists
- name: Check ECR Public repository exists
env:
SERVICE: ${{ steps.validate.outputs.service }}
run: bun .github/scripts/mirror-slim-image.ts ensure-repo
run: bun .github/scripts/mirror-slim-image.ts check-repo

- name: Mirror image
env:
Expand Down
6 changes: 3 additions & 3 deletions docs/adr/0026-slim-artifact-mirrors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ Some consumers cannot reach GitHub release assets or a single registry blob CDN.

## Decision

Publish each slim **image** to GHCR and AWS ECR Public (`public.ecr.aws/supabase/cli/<service>`), digest-preserving, via `mirror-slim-image.yml` and `.github/scripts/mirror-slim-image.ts`. Publish each slim **native** archive as an OCI artifact on the same two repositories under `:version-native-<target>` (not `:version-linux-*`, which are image platform tags). GitHub Releases remain the human HTTPS copy and `--clobber` on force.
Publish each slim **image** to GHCR and AWS ECR Public (`public.ecr.aws/supabase-registry/cli/<service>`), digest-preserving, via `mirror-slim-image.yml` and `.github/scripts/mirror-slim-image.ts`. Publish each slim **native** archive as an OCI artifact on the same two repositories under `:version-native-<target>` (not `:version-linux-*`, which are image platform tags). GitHub Releases remain the human HTTPS copy and `--clobber` on force.

`force=true` always requests the ECR copy, including when the image digest is unchanged (natives may have moved). The **image** destination digest gates `publish-release` once the dispatch token exists. Native ECR copy is best-effort (`continue-on-error`) and must not fail that release. Catalog sync consumes the image `service` / `version` / `digest` only; native tags ride in a separate `natives[]` payload field. Do not prune untagged GHCR or ECR manifests: already-shipped CLIs still pin old image digests until a catalog PR ships. Natives follow the moved tag immediately.

ECR Public tags are always mutable; `ecr-public create-repository` has no immutability flag. Reuse the existing `PROD_AWS_ROLE` in `supabase/cli`. Do not add a public S3 bucket or a second cloud vendor for this cut.
ECR Public tags are always mutable; `ecr-public create-repository` has no immutability flag. The repositories and the `ecr-push-public-cli` push role live in the central registry account, enrolled in `pulumi/registry/public/cli/cli.yaml` in `supabase/platform`; adding a service means adding it there first. Do not add a public S3 bucket or a second cloud vendor for this cut.

Client fail-through (ordered GHCR / ECR / GitHub candidates in the CLI and stack) is out of scope here and lands in a follow-up PR.

Expand All @@ -27,7 +27,7 @@ Client fail-through (ordered GHCR / ECR / GitHub candidates in the CLI and stack

## Rationale

ECR Public mirroring already exists for other CLI images and needs no new vendor. Native OCI on those same repos reuses `regctl` and that role. Extracting the workflow into bun scripts lets the copy, digest check, and native payload validation run in CI and locally.
ECR Public mirroring already exists for other CLI images and needs no new vendor. Native OCI on those same repos reuses `regctl` and the same push role. Extracting the workflow into bun scripts lets the copy, digest check, and native payload validation run in CI and locally.

## Consequences

Expand Down
Loading