Skip to content

Longhorn client-migration detach step holds on state left by an earlier run #5011

Open

Description

@devantler

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Evidence

Found by the merge-queue failure count in #4993.

  • 7 failed merge-group runs between 2026-10-11 16:51Z and 2026-10-12 02:06Z failed in the deploy job's Longhorn client-migration detach stage, which went into HOLD.
  • The first failure was on fix: allow volume clients on compute workers #4744, with a command-line "argument list too long" error. Later candidates, built on main, then held because maintenance ownership could not be proven, because state from an earlier run was still present, or because the volume inventory had changed.
  • Evidence basis: logs for 5 of the 7 runs (4 deploy, 1 heal); the other runs are grouped by time and step.
  • Examples: 38176558486, 38180721118, 38194496846.

Problem

One failed run left state behind that made the next candidates fail too, for about nine hours.

Expected behaviour

A failed or evicted candidate leaves nothing that stops the next candidate's migration step, or the heal path clears it. The step does not fail on the size of its own argument list.

Acceptance criteria

  • The "argument list too long" failure cannot recur, covered by a test with a large volume list.
  • State left by an evicted candidate is either cleared by the heal path or adopted safely by the next run, covered by a test.
  • A real ownership conflict still holds the step.

Size: small to medium. Part of #4979.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions