[stamp-build]

^{Gogo from botiverse opened this pull request through Stamp
View in Stamp →}

---

What

Holt x Orleans proposal P0-2: before a draining cluster instance lets its Durable Objects go, give every running clustered DO a bounded chance to flush.

On SIGTERM a cluster instance now:

1. Stops claiming, keeping its lease (Cluster::stop_claiming, new). The lease loop keeps renewing, so commits keep being acknowledged and no peer can claim the objects this node owns. Objects already running here keep serving requests. An object whose owner record names this node but that isn't open here still wakes here (may_keep). Claiming an unowned object, or one whose owner died, is still refused with the existing "this node is draining" error, which the server returns as DISCONNECTED.
2. Calls drain() on opted-in namespaces only. A namespace opts in with durableObjectNamespaces[].drain = true (new field, default false), so an existing app method named drain is never called by surprise. The node calls drain() on each running root actor of those namespaces, at most 32 per namespace at once. It waits for each method, and for what it wrote to be durable (its output gate), until a single deadline: cluster.drainTimeoutMs (default 6000; 0 skips the calls; values above 10000 are logged and clamped to 10000).
   • The method is looked up under the isolate lock but outside any request (actor_drain_method). It walks the prototype chain with GetOwnPropertyDescriptor and counts only a data property that is a function, so no app code runs. An accessor or a Proxy falls back to the request. An actor with no drain is therefore skipped without waiting behind its input gate.
   • The listeners keep serving during the drain, so requests in flight keep making progress.
   • A drain() still running at the deadline, or not yet started, is cancelled and counted as timed out.
3. Lets its objects go, as before: start_draining() (parks the lease loop and stops waking its own objects), abort_all(None) for each clustered namespace, deregister(). If a listener fails during the drain, these steps still run and the error is returned afterwards.

Each opted-in namespace logs one line: cluster drain; service = "<service>"; namespace = "<class>"; drained = N; skipped = N; failed = N; timed_out = N; ms = N. It is INFO normally and WARNING when anything failed or timed out, and it contains no object ids.

durableObjectNamespaces = [ ( className = "Room", uniqueKey = "...", drain = true ) ],

export class Room extends DurableObject {
  async drain() {
    // Called once when this instance drains, while it still owns the object.
    await this.ctx.storage.put('state', this.inMemoryState);
  }
}

How

• drain() is a WorkerInterface::CustomEvent (DrainCustomEvent, factory/worker-factory-actor.c++). It is started on the actor's own request through ActorContainer::start_request, the same path alarms and hibernated WebSocket events use. The event looks the method up on the instance (ExportedHandler::self), as [restore]() does, so ExportedHandler and the TS types are unchanged and classes without the method are skipped. After the method resolves it awaits waitForOutputLocks(), so "drained" means "durable in the bucket".
• New bridge functions worker_drain and actor_drain_method (server/bridge.rs), ActorNamespace::drains/drain_all and ActorContainer::drain (server/actor.rs), and leave_cluster/drain_actors in server/run.rs.

Not done (TODO in run.rs)

The optional step of releasing each drained object's own.json (node "") so peers claim it at once, instead of after up to 3 s of listing cache, is left as a TODO. Releasing at the end of a drain would let a peer open the next epoch while the actor still serves (stale) reads here until abort_all. Even after the abort, the eviction it starts (Owners::evict) still tiers the log into the bucket while the peer recovers from it, and nothing yet shows that race is safe.

Tests

• cluster/cluster.rs: a_node_that_stopped_claiming_keeps_its_objects_and_its_lease. After stop_claiming the node refuses new claims, its lease loop is not parked and still renews, and a peer routes to its object instead of claiming it.
• tests/cluster/cluster-test.mjs:
  • a draining owner's DO flushes in drain(), and the next owner sees it: a count kept only in memory is written by drain() and read back on the peer that takes over.
  • a drain() that does not finish is cut off at the deadline: the owner exits cleanly (code 0, no SIGKILL) about drainTimeoutMs (3 s in the test config) after SIGTERM, logs timed_out = 1, and what drain() committed before it hung is the next owner's.
  • a namespace that does not opt in is not drained: its class's drain() is not called, and the namespace logs no drain line.
  • an opted-in DO whose class has no drain() is skipped: logs skipped = 1 in under 1 s.
  • a drain() that throws fails alone, and the node still exits: logs failed = 1, exits with code 0, and what it wrote before throwing is the next owner's.
  • drainTimeoutMs = 0 calls no drain().
• server/cluster-test.rs: drainTimeoutMs is clamped to 10 s.
• cluster/cluster.rs: a_node_that_starts_draining_mid_claim_does_not_keep_the_object. A drain that starts while acquire_or_route awaits the bucket (between reading the owner record and advancing its epoch) makes the claim fail instead of returning Owned. acquire_or_route checks may_keep again right before returning Owned, and the test fails without that check.
• The stop_claiming unit test now also checks that the node still wakes its own object, and that it refuses to after start_draining.

Run locally (Linux x64, bazel test): //src/workerd/server/cluster:cluster_test (59 pass), //src/workerd/server/server:workerd-server_test, //src/workerd/server:server-test@, and //src/workerd/server/tests/cluster:cluster-test (32 pass, 1 pre-existing TODO; 235 s). --config=lint (clippy) is clean on the server and cluster crates, and tools/cross/format.py --check passes on the changed files.

Building a holt runtime binary

main has no botiverse build workflow, so this branch gets no binary from CI. .github/workflows/botiverse-build.yml exists only on the botiverse/* branches (botiverse/cluster-alarms, -v2, base-6780). It runs on push to botiverse/** and builds bazel build //src/workerd/server:workerd --config=release from scratch on a standard runner (a few hours), then uploads the artifact workerd-linux-x64-<sha>. To get a binary, push these commits plus that workflow to a botiverse/... branch. The binary then goes into holt's releases bucket as platform/<version>/workerd with SHA256SUMS (holt docs/overview.md).

Open questions / risks

• During the window, requests for new or dead-owner objects reaching a draining node get the existing "this node is draining" error. That was true before too, but the window is now up to drainTimeoutMs longer. The holt agent is to take the node out of Caddy before SIGTERM.
• Hibernated (evicted) actors are not running, so they are not drained. Their WebSockets are disconnected at the abort, as before. A running actor can use drain() to tell its WebSocket clients to reconnect.
• Facets are not drained separately; a root's drain() can call into its facets.
• Alarms that fire during the window run normally on objects running here. For an owned object not in memory, the claim is refused and the alarm is retried, as before.
• The drain must fit inside the supervisor's stop timeout, together with connection drain.

🤖 Generated with Claude Code