Skip to content

[Jetty] Closed a Workflow Stream from a System Nexus command of its chain. - #103

Open
moetemp wants to merge 23 commits into
moe/jetty/v3/t14-no-auto-closefrom
moe/jetty/v3/t15-system-nexus-close
Open

moetemp wants to merge 23 commits into
moe/jetty/v3/t14-no-auto-closefrom
moe/jetty/v3/t15-system-nexus-close

Conversation

@moetemp

@moetemp moetemp commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

This PR lets a Workflow close its own Stream with a System Nexus CloseStream command, and fixes the server findings of an independent review.

A Local Activity close runs before its Workflow Task commits. If the task failed after the close, the Stream stayed closed, and the retried task's write failed the run. A command commits with its task, and native Workers need no Activity for it.

What changed?

The close command.

  • CloseStream is a sync operation of the stream service. It routes like ReadRecords and DescribeStream.
  • The server gets the calling run from the operation's links. It writes the close as that run's Workflow Producer (wf:<run id>), with the request's result, failure and status. A caller that isn't an open run of the Stream's chain gets stream_not_owner. A run that a retry withdrew gets stream_fenced.
  • A start with no callback gets stream_invalid. Only a direct System Nexus call has no callback, so a direct caller must use the gRPC CloseStream.
  • The close Record has no epoch, so nothing withdraws it. The gRPC CloseStream and the close at the Run Chain end don't change.

Review fixes.

  • Who closes. A Local Activity can't close a Workflow-owned Stream. A second close of a closed Stream checks the closer first. A Workflow owner with no Workflow ref gets stream_invalid. A Workflow create of an open client Stream sets the owner, not only WorkflowOwned.
  • Reads. An empty filtered page scans past the scan cap to a match, the end or to_offset. Only a wake_only read with a callback parks, and a parked wake stores no Records. Other command reads return at once.
  • Reader floors. A Workflow page or wake sets the run's reader floor on a client Stream too. An append that would make the byte cap drop a floor's range fails with stream_too_large. The reader gets continue_as_new_suggested when its floor holds half of stream.maxStoredBytes. The readers check cancels the parked wakes of runs that closed or continued as new.
  • Run chains. A reset to an earlier run leaves the chain's Streams open, and the reset run's end closes them. A create with withdraw_run writes only the run's Superseded Record, with no epoch, as the crate does. The chain-end task skips the write to a first run with no Streams. An abandoned nested Stream node can be created or woken again.
  • Retention hold. A closed Standalone Activity is held until its Streams expire. On a standby, the hold writes with the passive transaction policy. Nested Streams are detached CHASM children, so their timers, wakes and readers check run after the host closes.
  • Shared vectors. Tests run the server's withdrawal rule and CompareEpochs on the crate's withdrawals.json and epochs.json.
  • Our Go lines wrap at 100 columns. The comments use literal wording and keep only the why.

Known limits.

  • A nested Stream that an earlier binary persisted isn't detached. Under a closed host, its tasks don't run until the next read or write of the Stream, and the hold keeps the host until then. For a dev server that ran an earlier binary, delete its database.
  • A close command in the run's final Workflow Task never runs, since the server drops the invocation of a closed run. The SDK awaits each close before the run ends.
  • Last still stops after the scan cap, as in the crate. A filtered last read can return fewer Records than asked when the matches are far back.
  • A command read that isn't a wake ignores wait and create_wait.

Why?

The review found that a Local Activity could close a Workflow's Stream before its task committed. It also found parked reads that could push an execution past the mutable state limit, and a reset that closed a chain's Streams. The new hold tests found that a nested Stream's tasks stopped when its host closed.

How did you test it?

Each behavior fix has a test that fails on 7dfc8a77 and passes at the head. The vector tests pass on both, and the standby policy has no test, since it needs an XDC setup. Unit tests of ./chasm/..., ./service/history/workflow/, ./common/api/, ./common/rpc/interceptor/ and ./service/frontend/ pass. The functional TestStreamTestSuite and TestStandaloneActivityTestSuite pass on SQLite. go vet is clean on the touched packages. The crate k3 and sdk-python j9 native lanes pass against a build of the head.

On the fork CI, every job passes at the head, fmt, golangci and Misc checks included. Those three failed since t14. The fixes are the go fix modernizers in read.go, the stale go.sum lines of the replaced go.temporal.io/api, and helpers that split ReadOrPark, Append and scan under revive's limits with no behavior change. The Mixed brain test failed once in its Omes scheduler_stress scenario, and passed on a rerun.

The superproject branch temporalio/internal-ai-prototypes 2026/10/ai198-streams pins the head, 9f912df3.

  • Unit Tests
  • Staging
  • End to End Tests

The server starts the operation only once the Workflow Task that issued it commits, so a failed
task closes nothing. The caller is the run in the operation's link, and the close goes in as its
Workflow Producer with no epoch.
A command's start always has the system callback and a direct call never has one. A direct call
waits for no Workflow Task to commit, so it gets stream_invalid and a pointer to the gRPC call.
A Local Activity writes before its Workflow Task commits, and a failed task can't take the close back. So only the close Activity or the System Nexus command closes.
The crate checks the owner first, so a caller that can't close gets stream_not_owner also after the close.
Outside a chain the Stream got the safety TTL with no chain check and no close. A Workflow create of a client Stream now changes the owner too, as the crate does.
A create has no task facts, so it withdrew a run's earlier tasks that a last read then hid. Its Superseded Record now has no epoch, as the crate's.
A page stopped after 10,000 entries of other Topics with no Record. A read with no wait took it for the end, and a parked read waited for a later append.
One append woke every parked read in one transaction and copied its batch into each. That could push the execution past the mutable state limit. Any other Workflow read now answers at once.
A wake whose create wait ended left the node abandoned, and every later create failed. A nested node has no execution of its own to replace.
The size cap dropped what a Workflow read from a client's Stream, and its replay then failed in a loop. An append that would drop a floor's range now fails, as on a Workflow Stream.
A wake outlived its run until the next write or its 7-day wait, so dead wakes filled stream.maxParkedReads.
The reset run carries the chain on, but the terminated run counted as the chain's end and closed the first run's Streams.
The active policy fails with NamespaceNotActive on a standby, and the queue retried the delete task with no end.
The namespace retention deleted the Activity and its Streams, though a closed Stream stays readable for its Retention. A closed host runs no Stream task, so the hold reads each Stream's expiry time.
A later run can't tell whether the first run has Streams, so every continue-as-new or retry chain end wrote the first run.
The server has its own copy of both rules, and nothing checked it against the crate's.
The stack already wrapped the chain-end code at 100. Two rows of upstream tables keep their gofmt alignment.
The helpers keep scan and ReadOrPark at or under their complexity before this round.
CHASM dropped the tasks of a Stream in a first run after continue-as-new or in a closed Activity. Its expiry, wake waits and readers check never ran, so the hold kept the host forever.
The fmt job runs go fix and fails on the loops it rewrites to slices.Backward and slices.Contains.
The replace points the module at api-go, so go mod tidy drops the upstream sums that Misc checks flags.
The golangci job failed on their complexity. The helpers keep the behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant