Skip to content

fix(rivetkit): refuse actions after onSleep with actor.restarting - #5876

Open
eersnington wants to merge 2 commits into
fix/drain-going-away-close-reasonfrom
fix/rivetkit-restarting-after-on-sleep
Open

eersnington wants to merge 2 commits into
fix/drain-going-away-close-reasonfrom
fix/rivetkit-restarting-after-on-sleep

Conversation

@eersnington

@eersnington eersnington commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Sleep grace accepted new actions for its whole duration. The TypeScript sleep hook runs the app's onSleep, then closes SQLite and clears runtime state such as vars. An action dispatched after that ran against a torn-down actor and failed with plain errors or rivetkit.internal_error. SleepFinalize refused with actor.stopping, which the client does not retry.

Now dispatch stays open only while the sleep hooks still run. After they finish, it refuses with the new actor.restarting error, and the client retries the action against the next generation.

Lifecycle state Before After
Started run run
SleepGrace, onSleep or connection disconnects running run run
SleepGrace, after those hooks run against a torn-down actor actor.restarting (503), client retries
SleepFinalize actor.stopping actor.restarting (503), client retries
DestroyGrace actor.stopping actor.stopping

The check reuses the existing core_dispatched_hooks counter, which counts the graceful cleanup and connection disconnects that emit_grace_events starts. No new state.

// rivetkit-core/src/actor/task.rs
LifecycleState::Started => None,
LifecycleState::SleepGrace if self.ctx.core_dispatched_hook_count() > 0 => None,
LifecycleState::SleepGrace | LifecycleState::SleepFinalize => {
	self.ctx.warn_work_sent_to_stopping_instance("dispatch");
	Some(ActorLifecycleError::Restarting.build())
}
LifecycleState::DestroyGrace => { /* actor.stopping, unchanged */ }
  • error.rs: adds ActorLifecycle::Restarting (actor.restarting, "Actor is restarting. Retry the request.") with status 503. It matches the TypeScript actorRestarting() error that the client already retries.
  • rivetkit-rust/engine/artifacts/errors/actor.restarting.json: generated error artifact.
  • sleep-sequence.md: documents when sleep grace admits work.
  • actor-sleep-db.test.ts: broadcast works in onSleep calls getCounts through a closure. The connection proxy read getCounts.bind as an action name, so the test failed before sleep started.
  • Behavior change: an action that arrives after the sleep hooks finish now runs in the next generation instead of the current one. Keep-awake work that waits for a later action during sleep grace no longer receives it in the same generation.

This is part 4 of 6 in a stack:

@railway-app

railway-app Bot commented Oct 13, 2026

Copy link
Copy Markdown

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

@claude

claude Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Review

The approach is sound. Reusing core_dispatched_hooks adds no new state, and returning a retryable actor.restarting (503) instead of the non-retried actor.stopping is the right fix. The tests cover both sides of the new boundary.

Issues

  1. Error artifact and registry look incomplete. CLAUDE.md says a new RivetError rewrites engine/artifacts/errors/*.json, and that you should run pnpm docs:gen-api so docs/api/error-registry.json matches. This PR only adds rivetkit-rust/engine/artifacts/errors/actor.restarting.json. engine/artifacts/errors/ has no actor.restarting.json, and docs/api/error-registry.json has no restarting entry. Check which directory cargo check writes to, and regenerate the docs. CI runs pnpm docs:check-api and fails if they differ.
  2. Possible race in the hook-count check. core_dispatched_hook_count() > 0 is read when each dispatch arrives. Check that emit_grace_events increments the counter before SleepGrace is observable by the dispatch loop. Otherwise an action arriving right at the start of grace gets actor.restarting before the hooks register. This is harmless because the client retries, but it is a spurious restart.
  3. Known behavior regression. Keep-awake work that waits for a later action during grace no longer receives it in the same generation. The PR calls this out. Confirm no driver test or example relies on it.

Tests

  • wait_for_grace_hooks_done is a polling loop (sleep(10ms) x 50). It is bounded and matches the neighboring wait_for_state. An event-driven wait (Notify or watch) would match CLAUDE.md better.
  • Add a test that DestroyGrace still returns actor.stopping, to lock in the table in the PR description.
  • The actor-sleep-db.test.ts change is fine.

Nits

  • The docs line says actor/restarting, while elsewhere the format is actor.restarting (<group>.<code>). Make these consistent.
  • The SleepGrace | SleepFinalize arm logs warn_work_sent_to_stopping_instance. That is now an expected path, since clients retry. Confirm the warn level will not be noisy.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant