Skip to content

Commit 1e976bd

Browse files
committed
Pull once more from the next registry when a pull limit runs out
Docker Hub does not count the HEAD of pick-base-image. So its pull limit can run out between the pick and the pull, and the leg failed with 429. - scripts/build-port makes the builder and runs the push and the export of each leg. When a pull fails with "toomanyrequests" or "429 Too Many Requests", the step runs once more with the same digest from the next registry. In a build, the 429 line must name the repository of the base image. Any other failure stops the step at once. - pick-base-image --after VALUE asks only the registries after the one of VALUE. - build-deploy.yml calls the script. The two builds start from actions/github-script, because a run: step gets no ACTIONS_RUNTIME_TOKEN for the gha cache. The export and the digest record take the base image that the push used. The buildkit pin keeps its "# tag:" marker. - tests/static/36-build-port.sh runs the script against a stand-in docker. Tests 35, 64 and 68 follow the new steps. docs/tests.md records each fault that each new assertion was seen to fail on.
1 parent 67fcc5a commit 1e976bd

12 files changed

Lines changed: 802 additions & 82 deletions

File tree

‎.github/workflows/build-deploy.yml‎

Lines changed: 36 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
# plan the version, the targets and the ports to build (scripts/plan-build)
44
# gate tests/run.sh static; the publish waits for it
55
# build one leg for each planned port, named "Build <arch>": inputs,
6-
# docker build, push by digest, evidence, image suite, digest record
6+
# builder, build and push by digest (scripts/build-port),
7+
# evidence, image suite, digest record
78
# publish tags on GHCR only (scripts/publish-ghcr)
89
# hub copies GHCR tags to Docker Hub (scripts/sync-hub)
910
# report raises or clears the issues build-<arch>, publish-ghcr, publish-hub
@@ -172,17 +173,6 @@ jobs:
172173
ref="$(scripts/pick-base-image)"
173174
echo "base_image=$ref" >> "$GITHUB_OUTPUT"
174175
175-
# The builder runs this image. A Docker Hub fault takes the same digest
176-
# from a mirror.
177-
- name: Pick the buildkit image registry
178-
id: buildkit
179-
env:
180-
# tag: buildx-stable-1
181-
BUILDKIT_IMAGE: docker.io/moby/buildkit:buildx-stable-1@sha256:cec9f139f45e93c5c69c60f8b07cfad9f43f4ef6b6a6cd917527fea5ff2e3dea
182-
run: |
183-
ref="$(scripts/pick-base-image --image "$BUILDKIT_IMAGE")"
184-
echo "ref=$ref" >> "$GITHUB_OUTPUT"
185-
186176
# The committed lists stay when generation fails. The digest record
187177
# says which lists the build used.
188178
- name: Generate the mirror lists
@@ -251,10 +241,14 @@ jobs:
251241
ALLOW_ANCHOR_DOWNGRADE: ${{ inputs.allow_anchor_downgrade || 'none' }}
252242
run: scripts/check-anchor-floor "$ARCH" "$ANCHOR"
253243

254-
- name: Set up Buildx
255-
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
256-
with:
257-
driver-opts: image=${{ steps.buildkit.outputs.ref }}
244+
# The builder runs this image. A Docker Hub fault takes the same digest
245+
# from a mirror. So does a pull limit that runs out after the pick.
246+
- name: Set up the builder
247+
id: buildkit
248+
env:
249+
# tag: buildx-stable-1
250+
BUILDKIT_IMAGE: docker.io/moby/buildkit:buildx-stable-1@sha256:cec9f139f45e93c5c69c60f8b07cfad9f43f4ef6b6a6cd917527fea5ff2e3dea
251+
run: scripts/build-port builder
258252

259253
- name: Log in to GHCR
260254
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
@@ -264,43 +258,40 @@ jobs:
264258
password: ${{ github.token }}
265259

266260
# No tag here. The digest goes to the publish job after every test below.
261+
# A pull limit that runs out after the pick builds once more from the
262+
# next registry, and base_image names the registry that the build used.
263+
# ⚠ A run: step gets no ACTIONS_RUNTIME_TOKEN, and the gha cache needs
264+
# it. github-script starts the script with the environment of an action.
267265
- name: Build and push by digest
268266
id: push
269-
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
267+
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
268+
env:
269+
TARGET_IMAGE: ${{ needs.plan.outputs.target_image }}
270+
BASE_IMAGE: ${{ steps.base.outputs.base_image }}
271+
IMAGE_VERSION: ${{ needs.plan.outputs.version }}
272+
SOURCE_COMMIT: ${{ github.sha }}
273+
BUILD_DATE: ${{ needs.plan.outputs.build_date }}
274+
TRANSPORT: ${{ steps.transport.outputs.transport }}
275+
SKIP_SERVERS: ${{ steps.lag.outputs.skip_servers }}
270276
with:
271-
context: ${{ github.workspace }}
272-
platforms: ${{ matrix.platform }}
273-
build-args: |
274-
IMAGE_VERSION=${{ needs.plan.outputs.version }}
275-
SOURCE_COMMIT=${{ github.sha }}
276-
BUILD_DATE=${{ needs.plan.outputs.build_date }}
277-
BASE_IMAGE=${{ steps.base.outputs.base_image }}
278-
TRANSPORT=${{ steps.transport.outputs.transport }}
279-
SKIP_SERVERS=${{ steps.lag.outputs.skip_servers }}
280-
outputs: type=image,name=${{ needs.plan.outputs.target_image }},push-by-digest=true,name-canonical=true,push=true
281-
cache-from: type=gha,scope=${{ matrix.arch }}
282-
cache-to: type=gha,mode=max,scope=${{ matrix.arch }},ignore-error=true
283-
provenance: mode=max
284-
sbom: true
277+
script: await exec.exec('scripts/build-port', ['push'])
285278

286279
# The databases that the build used, from the bootstrap stage in the
287280
# builder. The build args are those of the step above, so the stage is
288281
# the same one.
289282
- name: Export the databases that the build used
290-
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
283+
id: snapshot
284+
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
285+
env:
286+
DB_SNAPSHOT: ${{ runner.temp }}/dbsnapshot
287+
BASE_IMAGE: ${{ steps.push.outputs.base_image }}
288+
IMAGE_VERSION: ${{ needs.plan.outputs.version }}
289+
SOURCE_COMMIT: ${{ github.sha }}
290+
BUILD_DATE: ${{ needs.plan.outputs.build_date }}
291+
TRANSPORT: ${{ steps.transport.outputs.transport }}
292+
SKIP_SERVERS: ${{ steps.lag.outputs.skip_servers }}
291293
with:
292-
context: ${{ github.workspace }}
293-
platforms: ${{ matrix.platform }}
294-
target: dbsnapshot
295-
build-args: |
296-
IMAGE_VERSION=${{ needs.plan.outputs.version }}
297-
SOURCE_COMMIT=${{ github.sha }}
298-
BUILD_DATE=${{ needs.plan.outputs.build_date }}
299-
BASE_IMAGE=${{ steps.base.outputs.base_image }}
300-
TRANSPORT=${{ steps.transport.outputs.transport }}
301-
SKIP_SERVERS=${{ steps.lag.outputs.skip_servers }}
302-
outputs: type=local,dest=${{ runner.temp }}/dbsnapshot
303-
cache-from: type=gha,scope=${{ matrix.arch }}
294+
script: await exec.exec('scripts/build-port', ['snapshot'])
304295

305296
# --platform: the pushed index has an attestation manifest next to the
306297
# image, so a pull with no platform picks the platform of the runner.
@@ -345,7 +336,7 @@ jobs:
345336
ANCHOR: ${{ steps.anchor.outputs.anchor }}
346337
TRANSPORT: ${{ steps.transport.outputs.transport }}
347338
SKIP_SERVERS: ${{ steps.lag.outputs.skip_servers }}
348-
BASE_IMAGE: ${{ steps.base.outputs.base_image }}
339+
BASE_IMAGE: ${{ steps.push.outputs.base_image }}
349340
MIRRORLIST: ${{ steps.mirrors.outputs.mirrorlist }}
350341
BUILT_AT: ${{ needs.plan.outputs.build_date }}
351342
run: |

‎docs/design.md‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ Each rule below keeps one of these true for one way it could fail.
2323
| rule | failure it prevents | where |
2424
| --- | --- | --- |
2525
| The bootstrap stage starts from the official Arch Linux image, pinned by digest. No `FROM` names an image that this repository publishes. | A build from its own last output keeps each defect of that output, and nobody can rebuild it without trust in this project first. | `Dockerfile`, `tests/static/10-bootstrap-not-circular.sh` |
26-
| The build asks for the digest of the base image and of the buildkit image from `docker.io`, then `mirror.gcr.io`. For the base image, a Docker official image, `public.ecr.aws` is third. The same digest is the same bytes. A registry that answers with `ratelimit-remaining: 0` is passed over: its pull limit is used up. ⚠ The limit can run out between the probe and the pull. | One registry outage or one used-up pull limit stops every build. | `scripts/pick-base-image` |
26+
| The build asks for the digest of the base image and of the buildkit image from `docker.io`, then `mirror.gcr.io`. For the base image, a Docker official image, `public.ecr.aws` is third. The same digest is the same bytes. A registry that answers with `ratelimit-remaining: 0` is passed over: its pull limit is used up. A pull that fails with HTTP 429 or `toomanyrequests` runs once more, from the next registry at the same digest. ⚠ A second failure stops the leg, and the next run picks again. | One registry outage or one used-up pull limit stops every build. Docker Hub does not count the HEAD of the pick, so its limit can run out before the pull. | `scripts/pick-base-image`, `scripts/build-port`, `tests/static/36-build-port.sh` |
2727
| Each shipped `pacman.conf` sets `SigLevel = Required DatabaseOptional`. No SigLevel is `Never`. | A forged or unsigned package installs. Most ports do not sign their databases, so `DatabaseOptional` is necessary. | `rootfs/*/etc/pacman.conf`, `tests/static/30-signature-checking-on.sh` |
2828
| A port whose keys are not in `archlinux-keyring` has a keyring pin: the package, its sha256, each trusted fingerprint with its expiry. The package is in the tree. | The port deletes a superseded keyring package from its mirrors, and the build cannot get the trust root. | `bootstrap/keyrings/*.pin`, `bootstrap/any/usr/local/bin/install-port-keyring`, `tests/static/88-keyring-pins-vendored.sh` |
2929
| After populate, each pinned key that is not expired must be fully valid. | A key at marginal validity signs nothing that pacman accepts. The keyring installs, and the transaction commits nothing. | `install-port-keyring` |
@@ -135,18 +135,22 @@ The steps of a build leg, in order:
135135

136136
1. `scripts/binfmt-register ARCH`: the emulator, with the `F` flag.
137137
2. `scripts/pick-base-image`: the registry for the base image digest.
138-
3. `scripts/pick-base-image --image "$BUILDKIT_IMAGE"`: the registry for the buildkit image of the builder.
139-
4. `scripts/gen-mirrorlist ARCH`: exit 0 writes the lists in the checkout of the runner. Any other exit keeps the committed lists.
140-
5. `scripts/pick-transport ARCH`: `worker` or `oci`.
141-
6. `scripts/check-mirror-lag ARCH`: each `drop LIST SERVER` line becomes a `LIST=SERVER` pair of `SKIP_SERVERS`.
142-
7. `scripts/resolve-anchor ARCH`. ⛔ A port that no server answers for fails here: a red leg, never a green skip.
143-
8. `scripts/check-anchor-floor ARCH ANCHOR`. An anchor that no tag can carry fails here too.
144-
9. `docker buildx build`, pushed by digest to the target, with provenance (`mode=max`) and an SBOM. The cache is `type=gha` with `ignore-error=true`.
145-
10. The `dbsnapshot` target, exported to a directory.
138+
3. `scripts/gen-mirrorlist ARCH`: exit 0 writes the lists in the checkout of the runner. Any other exit keeps the committed lists.
139+
4. `scripts/pick-transport ARCH`: `worker` or `oci`.
140+
5. `scripts/check-mirror-lag ARCH`: each `drop LIST SERVER` line becomes a `LIST=SERVER` pair of `SKIP_SERVERS`.
141+
6. `scripts/resolve-anchor ARCH`. ⛔ A port that no server answers for fails here: a red leg, never a green skip.
142+
7. `scripts/check-anchor-floor ARCH ANCHOR`. An anchor that no tag can carry fails here too.
143+
8. `scripts/build-port builder`: the registry for the buildkit image (`pick-base-image --image "$BUILDKIT_IMAGE"`), its pull, and the builder that runs it.
144+
9. `scripts/build-port push`: `docker buildx build`, pushed by digest to the target, with provenance (`mode=max`) and an SBOM. The cache is `type=gha` with `ignore-error=true`.
145+
10. `scripts/build-port snapshot`: the `dbsnapshot` target, exported to a directory.
146146
11. `scripts/gen-evidence`, with `DB_SNAPSHOT` and `EXPECT_ANCHOR`.
147147
12. `tests/run.sh image` against the pushed digest.
148148
13. The artifacts `evidence-<arch>` (90 days) and `digest-<arch>` (7 days).
149149

150+
⭐ Steps 8 to 10 run once more when a pull fails on a used-up pull limit. The second try takes the same digest from the next registry: `pick-base-image --after`. A failure counts when the output has a line with `toomanyrequests` or `429 Too Many Requests`. In a build, that line must also name the repository of the base image, because a server of a mirror list can answer 429 too. Any other failure stops the step at once. The push names the base image it used, and the export and the digest record take that value.
151+
152+
Steps 9 and 10 start the script from `actions/github-script`. A `run:` step gets no `ACTIONS_RUNTIME_TOKEN`, and the `type=gha` cache needs it.
153+
150154
The build args of the Dockerfile: `BASE_IMAGE`, `TRANSPORT` (default `worker`), `SKIP_SERVERS` (default empty), `IMAGE_VERSION`, `SOURCE_COMMIT` and `BUILD_DATE`. Its stages: `bootstrap`, `dbsnapshot` and the final unnamed stage.
151155

152156
`digest-<arch>.json` has the keys `arch`, `platform`, `digest`, `anchor`, `transport`, `skip_servers`, `base_image`, `mirrorlist` (`generated` or `committed`) and `built_at`.

‎docs/measurements.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,11 @@ Measured facts only. Each row gives its date and its source: the command, the ru
169169
| image pins: 5, all current; tag push ages `archlinux:latest` 4 days, `registry:3` 17 days, `busybox:latest` 23 days | 2026-10-16 | `scripts/check-image-pins --max-age 14` |
170170
| `pick-base-image`: docker.io serves the digest of the base image, and of the buildkit image | 2026-10-16 | `scripts/pick-base-image`, and `scripts/pick-base-image --image` with the buildkit pin of `build-deploy.yml` |
171171
| `moby/buildkit:buildx-stable-1` at `sha256:cec9f139f45e93c5c69c60f8b07cfad9f43f4ef6b6a6cd917527fea5ff2e3dea`: docker.io and mirror.gcr.io answer 200 with that digest; public.ecr.aws (`docker/moby/buildkit`) and ghcr.io answer 404 | 2026-10-16 | `PROG=m; . scripts/lib.sh; reg_head REGISTRY REPO DIGEST` |
172+
| Docker Hub counts a GET of a manifest as a pull, and not a HEAD. Over the limit it answers HTTP 429: "You have reached your pull rate limit" | documented | docs.docker.com, Docker Hub usage and limits |
173+
| a registry over its request limit answers HTTP 429 with the error code `TOOMANYREQUESTS`; the docker client prints it as `toomanyrequests:` | documented | distribution/distribution `registry/api/errcode/register.go` |
174+
| the leg Build ppc64le: `setup-buildx-action` v4.4.1 ran `docker pull` of the buildkit pin, then `docker buildx create --driver docker-container --driver-opt image=<pin> --use`, then `docker buildx inspect --bootstrap` | 2026-10-16 | run 38930740850, the log of the job |
175+
| the same leg: `build-push-action` v7.4.0 ran `docker buildx build` with the 6 build args, the gha cache, `--attest type=provenance,mode=max,builder-id=<run URL>/attempts/1` and `--metadata-file`. The push added `--attest type=sbom,disabled=false`; buildx v0.38.0, BuildKit v0.33.1 | 2026-10-16 | run 38930740850, the log of the job |
176+
| the runner gives `ACTIONS_RUNTIME_TOKEN`, `ACTIONS_RESULTS_URL` and `ACTIONS_CACHE_SERVICE_V2` to a JavaScript action, and not to a `run:` step; buildx reads them for a `type=gha` cache | documented | actions/runner `NodeScriptActionHandler.cs`; docker/buildx `build/opt.go` |
172177
| pacman 7.1.0, a database that no server gives, through curl: `failed retrieving file` lines, then `failed to synchronize all databases (download library error)` | 2026-10-16 | `pacman -Sy` in the test container with `--network none`, against `file://` and `http://127.0.0.1:9` servers |
173178
| the same through an XferCommand: only `failed to synchronize all databases (error invoking external downloader)` | 2026-10-16 | the same, with an `XferCommand` |
174179
| a package that no server gives: through curl `warning: failed to retrieve some files`, then `failed to commit transaction (download library error)`; through an XferCommand `failed to commit transaction (error invoking external downloader)` | 2026-10-16 | `pacman -S`, the same way |

‎docs/operations.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -261,6 +261,7 @@ The workflows, the other scripts and the tests read the port table, and need no
261261
| the GHCR mirror is behind | the build drops it for that run (`drop LIST SERVER` in the log). The image still ships it. Nothing to do |
262262
| the Worker does not answer | `pick-transport` gives `oci`, and the build stage reads GHCR through pacman-oci. Nothing to do |
263263
| the base image registry does not answer | `pick-base-image` takes the same digest from `mirror.gcr.io`, or for the base image from `public.ecr.aws`. The buildkit image falls back to `mirror.gcr.io` too. Nothing to do |
264+
| a pull meets a used-up pull limit (HTTP 429, `toomanyrequests`) | `build-port` runs the step once more with the same digest from the next registry. The log says `met a used-up pull limit`. When the second try fails too, the port waits for the next run. Nothing to do |
264265
| `publish-hub` is open | the next run copies each missing tag. A failure there never touches GHCR. Check the Docker Hub secrets |
265266
| `registry-drift` is open | the job `hub` of the next build copies the missing and moved tags. A build with nothing to build runs `hub` too |
266267
| `status-down` or `pages-stale` is open | read the run of `status.yml` that the issue names, and start it again: `gh workflow run status.yml --ref main` |

0 commit comments

Comments
 (0)