Runtime "Sync Now"

[dgreer-dev]

flowctl raw sync-now --task X (and POST /v1/task-control/sync-now) forces an
immediate commit of a V2 materialization's open transaction and returns once that
transaction is committed and queryable in the destination. Closes #3281.

The workflow we want to enable is a user wanting to run analytics on the freshest
data available, regardless of any sync schedule (or other transaction-holding
reason). Essentially we want to allow flowctl raw sync-now ... && ./my-analyitcs.py
to be a safe and useful command. See below for specifics of the new command.

Design points worth your attention

• General commit barrier. Sync-now means "commit what you have, tell me when
  it's queryable", whatever is holding the transaction (schedule, min-duration
  floor, or just size). Worst case is a smaller transaction. "Nothing to do" is
  never an error: an idle task Acks IDLE, a capture or derivation
  NOT_APPLICABLE, both exit 0.
• One wait rule. Every outcome but IDLE/NOT_APPLICABLE resolves at the
  target transaction's Tail::Done, so N concurrent callers park on one commit.
• Server-side routing, no semantics in Go. The front door resolves shard
  zero with the caller's claims, proxies to the primary's peer via mayProxy,
  and relays 1:1 to the co-located sidecar.
• Two transports, one dialect. /v1/task-control/sync-now exists as a second
  way to force the sync. Calls the same code as flowctl, just an HTTP different path.
• Validation arm (first commit, independent of the runtime): an all-zero
  connector syncSchedule counts as unconfigured, so runtime-only pacing is
  expressible on every connector image already shipped; and a model-level
  syncSchedule without enable-runtime-v2 is now rejected rather than
  silently inert.

Verification

Rust e2e tests ensure transaction control works as advertised. Go tests stand up
two consumer test members against a stub sidecar, covering both transports
including the peer-proxied path. flowctl's response handling is covered against a
stub server over a real channel.

Also verified on a local stack against a schedule-held V2 materialization
(transcripts available). Highlights: three parallel flowctl pokes plus one
curl -N stream against one held transaction all exited 0 reporting identical
committed stats (one HELD_COLLAPSED, three ALREADY_CLOSING waiters);
destination row count equalled the pre-poke source count exactly on every run.

Example Usage

$ flowctl raw sync-now --task acmeCo/foo/materialize-bar
{"ack":{"outcome":"HELD_COLLAPSED","status":{"sourcedDocsTotal":"50","sourcedBytesTotal":"13691","openAgeMillis":"98573","headPhase":"Idle","tailPhase":"Done"}}}
{"done":{"committedDocsTotal":"50","committedBytesTotal":"13691","durationMillis":"99634"}}

Ack is immediate and carries the outcome plus a snapshot of the awaited
transaction. Done arrives when that transaction is committed and queryable in
the destination, with its committed stats. durationMillis is the
transaction's total open age, not the commit duration. Progress heartbeats
(~15s) are consumed for liveness but not printed unless --progress, since an
hour of them is a lot of lines unasked.

Any completed outcome exits 0, IDLE and NOT_APPLICABLE included - "nothing
to do" is success. Exit is non-zero for NOT_FOUND (not running here, or not
on the V2 runtime) and for --timeout elapsing.

Flags: --progress prints the heartbeats; --no-wait prints the Ack and hangs
up, which is harmless because the commit has already been forced; --timeout 10m bounds the whole invocation, which otherwise waits indefinitely and retries
leader restarts.

The same call over the REST transport, which is what the dashboard button will
use:

$ curl -N -H "Authorization: Bearer $TOKEN" \
    -d '{"task":"acmeCo/foo/materialize-bar"}' \
    https://$REACTOR/v1/task-control/sync-now
{"result":{"ack":{"outcome":"HELD_COLLAPSED","status":{...}}}}
{"result":{"done":{"committedDocsTotal":"49",...}}}

Commits

1. Types and basic verification
2. Wiring runtime-next
3. Go -> runtime pathway
4. e2e tests

[williamhbaker]

Some directional comments! I didn't get into the gritty details just yet since I am hoping things can be simplified quite a bit.

[dgreer-dev]

Pushed a commit with the four resolved convo fixes.

• Dropped --progress, --no-wait, --timeout. flowctl now prints nothing and exits 0; the exit status is the whole contract.
• SyncNowResponse is down to three empty messages: Ack, Heartbeat, Done. Status and the outcome enum are gone, and so are Done's committed stats (same argument as Status).
• "Nothing to await" needs no label of its own: the leader acks and Done follows immediately, which is what IDLE and NOT_APPLICABLE already did.
• Dropped the validation error on syncSchedule without enable-runtime-v2.

[williamhbaker]

Looking pretty good with the updates. I had a couple of pending comments that I think are still relevant, which will hopefully show up below this...