[pinotelio]

Fixes the flake of internal/engine TestEngineHealthReportsTheAppliedEpoch: it failed 2 of 3 local runs and again in #49's engine-backed run, with LastApply() = {Epoch:0 RevisionID: Status: ...}.

Root cause

Every failure of TestEngineHealthReportsTheAppliedEpoch was its last step: one read of /fleet after waiting for /health, which answered the zero record (LastApply() = {Epoch:0 RevisionID: Status: ...}). The engine orders an apply that way on purpose (Crewlet's internal/engine/reconcile.go, Reconciler.apply: "THE OUTCOME REACHES THIS NODE'S OWN PROGRESS FIRST, before the record that tells its peers"). A successful apply moves the node's own progress, which /health's applied epoch reads, then writes its boot row to the node's own store (Reconciler.serve), and only then writes the record its peers read in /fleet to the coordination store (Reconciler.record, through RecordApply): one write to the node's store and one to the coordination store later. A single read of /fleet made as soon as /health reports the epoch can land between the two and find the record not yet written. It is not a product defect: every reader of that record in the operator already allows for the lag and keeps more for one look, never less (failureAccount says the record may not hold the failure yet, narrowPlacement waits for the record to name the revision, and keepServing reads a zero record as a node that has not said what it runs).

Fix (test only)

• The case waits for the /fleet record as it waits for /health, with the same bounded eventually.
  • A record behind the apply's epoch (none yet, or an earlier apply's) is the lag, and is waited for.
  • A record at the apply's epoch or later that is not the expected one fails at once, naming itself (recordSettled).
• TestARecordOfTheApplyIsWaitedForAndAWrongOneFails pins that decision deterministically. It covers a record not yet written, an earlier apply's, the apply's own, another revision at the apply's epoch, and a failure at it. It fails if any record at the epoch is accepted, or if a wrong one is waited on instead of failing.
• What this PR does not claim: that the old single read fails reliably. Whether a read lands between /health and the record is timing.

Checks

• Against an engine built from crewlet main 0d7ea94840, -race -p 1:
  • TestEngineHealthReportsTheAppliedEpoch, 3 runs in a row: ok;
  • ./internal/engine/: ok.
• golangci-lint (v2.14.0, go1.27.2): 0 issues.

The commit message says the gap is "two writes to the coordination store"; it is the node-store write of the boot row and the coordination-store write of the record, as above (corrected here rather than by amending).