Skip to content

feat: revert the temporary no-cache on docker builds once every repo is rebuilt #953

Open

Description

@bedatty

Request Type

Refactor / simplification (revert of a temporary mitigation)

Affected Workflow

Infrastructure (gitops-update, build, helm-update-chart, dispatch-helm) — .github/workflows/build.yml and src/build/docker-build-ts/action.yml

Problem / Motivation

no-cache: true was added to the docker/build-push-action steps in .github/workflows/build.yml (verification build and build-and-push) and in src/build/docker-build-ts/action.yml (build-and-push) as a temporary mitigation.

Reason: with cache-from: type=gha, BuildKit replays the cached apk/apt layer, so a rebuild ships the same vulnerable ca-certificates package instead of picking up the fixed one (https://emudhra.we-have-a-problem.com/). Forcing no-cache guarantees every published image resolves packages fresh.

This is a deliberate trade-off: every build now pays the full build time, across every repository in the organization that consumes these workflows. It must not stay in place longer than necessary.

Proposed Solution

Once all consumer repositories have been rebuilt and redeployed with a clean ca-certificates, revert the mitigation:

  1. Remove no-cache: true and its # TEMPORARY: comment block from the two steps in .github/workflows/build.yml.
  2. Remove no-cache: true and its # TEMPORARY: comment block from the build-and-push step in src/build/docker-build-ts/action.yml.
  3. Release through the normal tier promotion so consumers get the cache back.

cache-to: type=gha,mode=max was intentionally kept while the mitigation is active, so the clean layers are already exported — the first build after the revert starts from a healthy cache rather than a stale one.

Alternatives Considered

  • Busting the cache via a changing build-arg (e.g. CACHEBUST) — same cost, more machinery, and it leaves a permanent input behind.
  • Pinning the fixed ca-certificates version in each Dockerfile — correct long term, but it needs a change in every consumer repository and does not help images already in flight.

Example Usage

# Current (temporary) state in build.yml
- name: Build and push Docker image
  uses: docker/build-push-action@...
  with:
    # TEMPORARY: the gha cache replays the old apk/apt layer, so a rebuild keeps
    # the vulnerable ca-certificates. Revert once every repo has been rebuilt and
    # redeployed. See the tracking issue.
    no-cache: true
    cache-from: type=gha
    cache-to: type=gha,mode=max

# Target state after the revert
- name: Build and push Docker image
  uses: docker/build-push-action@...
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

Would This Be a Breaking Change?

No — fully backward compatible. No input, output or default changes for callers; only build duration is affected.

Checklist

  • I searched existing issues and this is not a duplicate.
  • This feature aligns with the repository's goal of providing reusable, organization-wide workflows.

Additional Context

Blocked until every repository consuming build.yml / docker-build-ts has been rebuilt and redeployed. Close only after that sweep is confirmed.

Activity

  1. bedatty commented on Oct 21, 2026

    @bedatty
    ContributorAuthor

    The mitigation landed in #954 with two flags, not one — the revert has to remove both from the same three steps:

    • no-cache: true — ignores the imported gha cache so the apk/apt step actually re-runs.
    • pull: true — re-resolves FROM. no-cache alone does not touch the base image, so a ca-certificates shipping inside the base would have survived it.

    Also worth weighing when scheduling this: in build.yml the verification build exists so its layers land in the gha cache and the multi-arch push build reuses them. With no-cache on both, that reuse is gone and every job compiles the image twice from scratch — the cost of leaving this in place is roughly double build time per component, across every consumer repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or improvement request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions