Skip to content

Talos image factory transients fail the deploy's cluster update #5013

Open

Description

@devantler

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence

Found by the merge-queue failure count in #4993.

  • 6 failed merge-group runs, on 2026-09-20 and 2026-10-12, failed in the deploy job's Talos cluster update because the Talos image factory was unreachable or slow: a TLS handshake timeout or deadline while registering the schematic, or a failed snapshot image download.
  • Evidence basis: logs for all 6 runs.
  • Examples: 35488969077, 38266777124, 38306469705.

Problem

A slow third-party service evicts a candidate and starts a heal deploy, although nothing in the candidate needed a new image.

Expected behaviour

A transient image-factory error during cluster update is retried within a fixed bound, and only a lasting failure fails the deploy.

Acceptance criteria

  • Decide where the retry belongs: the deploy's call into KSail, or KSail itself (file the KSail issue if it is there).
  • A transient timeout or download failure is retried with backoff, within a fixed bound; a lasting failure still fails and names the service.
  • A test covers one transient failure followed by success.

Size: small. Part of #4979.

Activity

  1. added theissue type on Oct 17, 2026
  2. moved this to 📥 Backlog in 🌊 Project Boardon Oct 17, 2026
  3. devantler commented on Oct 24, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    Refinement, 2026-10-24: half of this has a root-cause fix in KSail.

    • The schematic-registration timeouts were diagnosed in Identify the hosted Image Factory registration transport failure ksail#7707 (closed 2026-10-13): on hosted runners the registration's TLS handshake stalls for large writes, while split writes succeed with the normal TLS settings. fix: preserve secure factory registration on constrained network paths ksail#7708 (merged 2026-10-13) changes only the registration client's write sizing, keeping verified TLS. The platform now runs KSail 7.203.18 in CI and deploys, well after that fix.
    • That answers this issue's first criterion for the registration case: the fix belongs in KSail, and it removes the stall instead of retrying around it.
    • Still open here: the other failure this issue names, a failed snapshot image download from the image factory during cluster update. Nothing I found retries or explains that one yet, so the second and third criteria remain for it.

    No change was made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions