Put the native stream commits on the unified Core. by moedash · Pull Request #4 · moedash/sdk-rust

moedash · 2026-09-16T17:17:05Z

This PR replays the native stream commits onto the unified Core as linear cherry-picks.

What changed?

The content of Let a workflow subscribe to a stream and append records to it. #6 and Delivered server-side stream ranges to workflows. #7, cherry-picked onto Merged the current Core into the external stream branch. #3. The stream machines, core_tests/streams.rs and the stream proto files match moe/AI-198-core-3-delivery byte for byte. The lang protos differ only in the external chain's lines.
The native entries share their numbers with the split chain: deliver_stream_records at 21, subscribe_stream at 29 and append_stream_records at 30, around the external family. unsubscribe_notification_channel at 32 and workflow_stream_channels at 33 follow them, once each.
The channel machines are Carried the external stream task and replay wake fixes. #2's copies, the subscribe and the unsubscribe alike. Delivered server-side stream ranges to workflows. #7 carries its own mirrors of both, and this branch does not cherry-pick them, so one machine answers each command.
Lang appends with AppendStreamRecords and subscribes with SubscribeStream, which carries start_position. Ranges arrive as DeliverStreamRecords jobs. Each StreamRecord keeps its kind, producer id, attempt and sequence through Core.
A recorded range arrives in the same activation as its task across History page boundaries. A replayed append or subscribe is checked against its recorded event.
HistoryForReplay::with_stream_slices lets a language replayer hand Core the records it fetched. A missing re-supply fails as WFMachinesError::MissingRecords, a worker failure and never nondeterminism.
Merged the current Core into the external stream branch. #3 is merged forward, so this Core carries the native stream_slices on the poll response and the external chain's channel notifications on the scheduled event.
Through Merged the current Core into the external stream branch. #3 as well, the WorkflowStreamChannels report and the subscribe deferred to the completion that ends the Workflow Task.
Through Merged the current Core into the external stream branch. #3 as well, the UnsubscribeNotificationChannel command and machine, issued for a channel that left the reported set ahead of any subscribe on the same completion.
Through Merged the current Core into the external stream branch. #3 as well, the vendored unsubscribe command, event and failed cause, and ChannelSubscriptionInfo with DescribeWorkflowExecutionResponse.channel_subscriptions.

Review the stream content on #6 and #7. This PR is where the two lineages meet.

Part of AI-198 (epic AI-37).

Why?

One Core that speaks both prototypes means one Python build runs every provider. The merge base with #7's branch is upstream, so merging it would bring the whole native series in a second time, and cherry-picks avoid that. Changes the two chains share reach this branch by merging #2 forward.

How did you test it?

The full cargo suite ran, both stream test families included, and the workspace unit suite, cargo fmt --check and cargo test-lint ran again on the head that carries the deferred subscribe and the unsubscribe. The replay worker was fed a read-then-publish history with its slices, with a wrong slice and with none, and a missing re-supply fails before the first activation. The Python bridge builds against this head. moedash/sdk-python#6 pins it and runs its whole suite on it.

Unit Tests
Staging
End to End Tests

History records the offsets a task consumed and never the payloads, so the server sends the bytes on the poll response: untagged for the task about to run, and tagged with a WorkflowTaskCompleted event id when re-supplying what an earlier task consumed. Core partitions the two and emits recorded ranges in event order before the live one, so a replaying workflow observes them exactly as it did the first time. An empty range is delivered rather than dropped: a task where the subscription saw nothing is a fact replay has to reproduce. The Rust SDKs have no stream API, so they fail loudly on this job instead of ignoring it. The server has already recorded the range as consumed and will not send it again, so dropping it would lose data silently.

The command has a history event, so it fits the matching every SDK's replay depends on: commands are popped from a queue as command-generated events arrive, and one producing no event would put that out of step. The machine never resolves. The event records the subscription and hands nothing back; the ranges arrive later as their own activation jobs.

The reporter panics on any machine name it has no visualizer for, so every state machine has to be listed there.

The range a task consumed is recorded on the completion that closes it, and the machines look ahead to that event while replaying the task. An update cut at a WFT started event left the completion on the retained tail or on an unfetched page, so the range arrived one activation late.

A publish is matched on its stream and message count, a subscription on its stream. Replay sends no commands, so a reissued command that differs from the record would otherwise be accepted and the workflow's state would diverge silently.

A completion that records consumed stream cursors ends its task sequence. Such a task ran as one activation live, and folding it into a heartbeat chain would hand several ranges over at once on replay.

Ranges for one task are handed over sorted by stream and offset, so a workflow waiting on several streams sees the same order live and on replay. A re-supplied slice has to cover exactly the recorded offsets, and a range that observed nothing is rebuilt from the cursor rather than demanded from the server.

The external stream family, developed alongside this one, takes 17 to 20 in the activation job and 23 to 28 in the command. Both trees now encode the native stream messages at 21, 29 and 30, so an artifact generated from either decodes correctly against the other.

…lary. The api names an entry a record and carries its kind, producer, attempt and sequence on it, so the vendored stream protos follow the api head and lang appends with AppendStreamRecords and receives DeliverStreamRecords. Core passes each record through untouched.

History records only the offsets a task consumed, so a language replayer that fetched the records from the stream service needs a way to hand them to Core with the history. The replay worker puts them on its synthetic poll response, so the ordinary delivery path and identity checks run unchanged.

The bytes for a recorded range only travel on the response that carries the task, so a sticky task, or a sticky legacy query, handed to a worker that no longer holds the run and fetched the history itself cannot be replayed. Failing when the lookahead sees the range keeps the workflow off less input than it had, and lets a legacy query go unanswered so the server retries it on the normal queue, where the records travel with it.

A recorded range with content and no bytes for it is the worker's failure to reconstruct the run, not the workflow's nondeterminism, so it is its own error kind and treated like a failed history fetch: the task fails as an unhandled worker failure and a legacy query goes unanswered, so the server retries both where the records travel. The broken run is evicted even when the query failure is withheld, so the retry starts from history.

Both sides carried the stream protos: the unified branch from the repairs branch, this one from the lookahead branch's earlier cut. The seven files are the api branch's diff on this Core's own api tree, and the protos module keeps this branch's arms, which already cover the incoming ones.

A subscription's explicit start offset and an unnamed append's resolved stream both went unchecked, so a replay that asked for something else passed. The offset check skips a repeat subscribe, which the server records at the cursor rather than at what the command asked for.

Both command enums are uninhabited, so neither arm runs. Nondeterminism is the wrong label for an internal invariant when the rest of the series works to keep worker failures out of that bucket.

A task owed two streams and sent one came out as nondeterminism, which a worker configured for it fails the execution over. Both sides of that comparison come from the server, so no arm of it is the workflow's fault.

The flag that says a completion consumed a range was written into the two that carry across the whole scan, which only works while every path after it returns. A local one says what it means on its own.

The fetch costs a page for any run whose page boundary lands there. A subscription made through the stream service records no event, so there is no telling a stream run from any other before the ranges arrive.

The blank comment line folded into the line above it, so the summary and the paragraph under it ran together.

Reordering two arms of a match on distinct variants changes nothing and shows up as a deletion on a branch whose claim is that it only adds.

Comparing it is only right for a run's first subscribe to a stream, and a subscription made through the stream service leaves no event, so which one is first cannot be told. Failing a sound run costs more than the drift.

The append event carries the exclusive end offset instead of a count, so the three range-carrying messages read the same way. The command fields say name rather than id, which is what a Workflow actually addresses.

The lang command now says stream_name for an append and stream_name_or_id for a subscribe, so one vocabulary runs from lang through to History. The append machine reads the batch size off the event's offset range.

The delivery tests build their own commands and histories, so they name the fields directly. One recorded append passed a count where the event now wants an end offset.

# Conflicts: # crates/protos/protos/local/temporal/sdk/core/workflow_activation/workflow_activation.proto # crates/sdk-core/src/worker/workflow/managed_run.rs

The main merge introduced the released headings above them and the resolution left the entries inside 0.8.0. The checkpoint requires additions under Unreleased.

The api adds a StreamStartPosition message and a start_position field on the subscribe command, at a new field number, so a Workflow can ask for the earliest record, the tail or the last N.

Lang can now ask for the earliest record, the tail or the last N instead of a negative offset. Core only forwards it: the server resolves it and records the offset, which is all replay matches against.

# Conflicts: # crates/sdk-core/src/protosext/mod.rs # crates/sdk-core/src/worker/workflow/history_update.rs # crates/sdk-core/src/worker/workflow/mod.rs

# Conflicts: # crates/protos/protos/local/temporal/sdk/core/workflow_activation/workflow_activation.proto # crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs

# Conflicts: # crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs

# Conflicts: # crates/sdk-core/src/protosext/mod.rs # crates/sdk-core/src/worker/workflow/history_update.rs # crates/sdk-core/src/worker/workflow/mod.rs

# Conflicts: # crates/sdk-core/src/worker/workflow/machines/transition_coverage.rs # crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs # crates/sdk-core/src/worker/workflow/mod.rs

Carries #2 7006981 through #3 093c55e: the five channel requests take a common.v1.Execution and both linked_to fields become Execution. The native stream protos are untouched.

Carries #2 677d468 through #3 2a516e1: the execution and linked_to fields reuse the old numbers and nothing is reserved. The native stream protos are untouched.

moedash · 2026-10-03T09:46:33Z

Replaced by #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, #19, #20, #21, #22 and #23.

Same content, split into 16 PRs in the v3 series: the notification channel first, then the streaming interface, then native streams and the rest. The branch stays as a pin.

moedash force-pushed the moe/AI-198-core-unified branch from ddd150d to 104a15d Compare September 16, 2026 21:52

moedash force-pushed the moe/AI-198-core-all branch 2 times, most recently from bdf72f6 to d9c8bb4 Compare September 16, 2026 22:02

moedash force-pushed the moe/AI-198-core-unified branch from 104a15d to f4d6ee1 Compare September 16, 2026 22:07

moedash force-pushed the moe/AI-198-core-all branch from d9c8bb4 to 4f4c8a4 Compare September 16, 2026 22:07

moedash force-pushed the moe/AI-198-core-unified branch from f4d6ee1 to ce28a0e Compare September 16, 2026 23:13

moedash force-pushed the moe/AI-198-core-all branch 5 times, most recently from e9b3c4a to 02c0792 Compare September 17, 2026 06:24

moedash added 12 commits September 18, 2026 17:43

Let a workflow publish to a stream from lang.

287ba43

Replayed recorded stream ranges before matching commands.

338beea

Registered the stream machines with the coverage reporter.

a1c15f4

The reporter panics on any machine name it has no visualizer for, so every state machine has to be listed there.

Gave a data-only stream task its own replay activation.

a02ccce

A completion that records consumed stream cursors ends its task sequence. Such a task ran as one activation live, and folding it into a heartbeat chain would hand several ranges over at once on replay.

Covered the stream delivery guards and each range's activation.

e32ea19

Added the changelog entry for the native stream commands and job.

419b548

moedash force-pushed the moe/AI-198-core-all branch from 02c0792 to 419b548 Compare September 19, 2026 00:51

moedash added 6 commits September 21, 2026 01:48

moedash added 16 commits September 25, 2026 13:50

Named the unreachable stream machine responses fatal.

7622031

Both command enums are uninhabited, so neither arm runs. Nondeterminism is the wrong label for an internal invariant when the rest of the series works to keep worker failures out of that bucket.

Failed a re-supply that disagrees with History as the worker's.

ab70e2e

A task owed two streams and sent one came out as nondeterminism, which a worker configured for it fails the execution over. Both sides of that comparison come from the server, so no arm of it is the workflow's fault.

Scoped the data-only task flag to the task it describes.

473a1a4

The flag that says a completion consumed a range was written into the two that carry across the whole scan, which only works while every path after it returns. A local one says what it means on its own.

Said why the extra boundary page cannot be fetched more narrowly.

a7c3b1c

The fetch costs a page for any run whose page boundary lands there. A subscription made through the stream service records no event, so there is no telling a stream run from any other before the ranges arrive.

Restored the comment break the append proto lost.

93d7678

The blank comment line folded into the line above it, so the summary and the paragraph under it ran together.

Kept the command match arms in their original order.

20064ee

Reordering two arms of a match on distinct variants changes nothing and shows up as a deletion on a branch whose claim is that it only adds.

Merged the repairs and unified review round into the all branch.

0c45d00

Dropped the subscribe start offset check as unsound.

618e8a0

Comparing it is only right for a run's first subscribe to a stream, and a subscription made through the stream service leaves no event, so which one is first cannot be told. Failing a sound run costs more than the drift.

Renamed the stream command and event fields.

04ce5c2

The append event carries the exclusive end offset instead of a count, so the three range-carrying messages read the same way. The command fields say name rather than id, which is what a Workflow actually addresses.

Renamed the lang stream command fields to match the wire.

748f1c3

The lang command now says stream_name for an append and stream_name_or_id for a subscribe, so one vocabulary runs from lang through to History. The append machine reads the batch size off the event's offset range.

Carried the stream field rename into the delivery tests.

32ec55d

The delivery tests build their own commands and histories, so they name the fields directly. One recorded append passed a count where the event now wants an end offset.

Merged the current upstream Core into the all branch.

d2b9dac

# Conflicts: # crates/protos/protos/local/temporal/sdk/core/workflow_activation/workflow_activation.proto # crates/sdk-core/src/worker/workflow/managed_run.rs

Moved the stream changelog entries back under Unreleased.

a55d2d9

The main merge introduced the released headings above them and the resolution left the entries inside 0.8.0. The checkpoint requires additions under Unreleased.

Vendored the api's subscribe start position.

6da86ea

The api adds a StreamStartPosition message and a start_position field on the subscribe command, at a new field number, so a Workflow can ask for the earliest record, the tail or the last N.

Passed the subscribe start position from lang to the server.

e03957e

Lang can now ask for the earliest record, the tail or the last N instead of a negative offset. Core only forwards it: the server resolves it and records the offset, which is all replay matches against.

Merge commit 'f62993b1' into to-all

6ae3829

# Conflicts: # crates/sdk-core/src/protosext/mod.rs # crates/sdk-core/src/worker/workflow/history_update.rs # crates/sdk-core/src/worker/workflow/mod.rs

This was referenced Oct 1, 2026

Merged the current Core into the external stream branch. #3

Closed

Delivered server-side stream ranges to workflows. #7

Closed

moedash added 8 commits October 1, 2026 18:13

Merge commit '736ecc8f' into to-all

899511e

Merge commit '741ff2bb' into to-all

614cb65

# Conflicts: # crates/protos/protos/local/temporal/sdk/core/workflow_activation/workflow_activation.proto # crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs

Merge commit '1de8b920' into to-all

5b4b857

# Conflicts: # crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs

Merge commit 'ba05a851' into to-all

d7684be

# Conflicts: # crates/sdk-core/src/protosext/mod.rs # crates/sdk-core/src/worker/workflow/history_update.rs # crates/sdk-core/src/worker/workflow/mod.rs

Merge commit 'c3d0a4b3' into to-all

c3b4b35

Merge commit '12b429e9' into to-all

67426be

# Conflicts: # crates/sdk-core/src/worker/workflow/machines/transition_coverage.rs # crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs # crates/sdk-core/src/worker/workflow/mod.rs

Merged the execution-addressed unified core into to-all.

c017e35

Carries #2 7006981 through #3 093c55e: the five channel requests take a common.v1.Execution and both linked_to fields become Execution. The native stream protos are untouched.

Merged the renumbered unified core into to-all.

0871616

Carries #2 677d468 through #3 2a516e1: the execution and linked_to fields reuse the old numbers and nothing is reserved. The native stream protos are untouched.

moedash closed this Oct 3, 2026

Navigation Menu

Put the native stream commits on the unified Core. - #4

Put the native stream commits on the unified Core.#4
moedash wants to merge 42 commits into
moe/AI-198-core-unifiedfrom
moe/AI-198-core-all

moedash commented Sep 16, 2026 •

edited

Loading

Uh oh!

moedash commented Oct 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

moedash commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

How did you test it?

Uh oh!

moedash commented Oct 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

moedash commented Sep 16, 2026 •

edited

Loading