Repository navigation
Durable Object alarms in cluster mode (one host, several processes) - #1
Closed
stamp-build[bot] wants to merge 29 commits into
Closed
stamp-build[bot] wants to merge 29 commits into
stamp-build[bot] wants to merge 29 commits into
Conversation
This defines how the new feature is configured and exposed on the network.
RAII wrapper around Linux OFD locks (fcntl F_OFD_SETLK) for per-fd file locking needed by the cluster DO ownership protocol. OFD locks are per open-file-description rather than per-process, which is essential since a single workerd process manages many DOs concurrently. Includes tryLock() (non-blocking), verifyHeld() (for NFSv4 lease verification), and move semantics. Non-Linux platforms fail at runtime rather than compile time so builds still succeed when cluster mode is unused. This code (and commit message) was entirely written by Opus 4.6.
Implements ClusterRegistry, a capnp::VatNetwork for multi-node cluster RPC. Each node generates an X25519 keypair as its identity, binds a listener (Unix socket or ephemeral TCP port based on CIDR), and registers itself in a shared directory. Includes: - cluster.capnp: VatId and stub Level 3/4 types for VatNetwork params - X25519PublicKey: small wrapper with hex encoding and HashMap support - ConnectionImpl: per-peer Connection with MessageStream, idle timeout, and refcounted outbound caching - Handshake protocol: raw 32-byte public key exchange before RPC starts - Peer discovery: lazy directory scanning with mtime-based liveness - CIDR/IP mode: getifaddrs-based IP matching, OFD-locked registry file with periodic mtime heartbeat - Unix mode: socket-file-as-registry-entry, connect-based liveness The end-to-end RPC tests are present but currently failing; the non-network tests (keypair generation, self-bootstrap, peer discovery, self-connect) all pass. This code (and commit message) was originally written by Opus 4.6 but the code sucked and was heavily refactored by my human hand.
Implements ClusterLockManager, which manages the lock files that serialize ownership of Durable Objects across a workerd cluster. Each lock file is named after the actor ID and contains the owner's 32-byte X25519 public key (or is empty if unowned). acquireOrRoute() encapsulates the full protocol: - If the lock file is the expected 32-byte size, read the owner key and look it up in the ClusterRegistry. If the owner is live (or might be), return a WorkerdDebugPort capability via clusterRpc.bootstrap() — the RpcSystem short-circuits self-routing without going through the VatNetwork. - Otherwise (empty file or confirmed-dead owner), try to acquire an exclusive OFD lock. If granted, ftruncate, write our identity, fsync, and return an OwnedLock. If refused, retry with exponential backoff and jitter. No checksum is needed on the lock file payload: a single pwrite() of 32 bytes is atomic with respect to concurrent readers (both on local filesystems and over NFS, where it's a single RPC to the server). The writer's sequence (ftruncate → pwrite → fsync) means readers see either size 0 or the full 32-byte key, never a partial write. This is a deviation from the original spec, which called for a CRC32. OwnedLock is an RAII handle whose destructor truncates the file and releases the lock, returning the DO to the unowned pool. verifyStillOwned() is provided for the NFSv4 lease check that the SQLite VFS will call on every file open (Commit 4). This code (and commit message) was originally written by Opus 4.6 with a fair amount of manual cleanup.
This callback is invoked immediately after `open()` opens a file. In NFS mode we'll use this hook to verify the new open was opened on the same NFS lease as the registry lock.
In cluster mode, all members of the cluster need to use the same key for channel tokens, so they can exchange them.
If the actor ID was derived from a name, it's important to pass the name along with the ID so that `ctx.id.name` works correctly.
(This is the commit that wires everything up, but this commit message describes the overall series.) workerd has always been intended to be something you can run in production, in order to self-host Workers outside of Cloudflare. For many use cases it does in fact work for this. But, there has been one missing piece: Durable Objects. workerd has always supported running Durable Objects for the purpose of local testing, but only within the scope of a single instance running on a single thread. Any Durable Objects created would run within that same workerd instance. But in production, you probably want to utilize more than one thread on one machine. For stateless Workers, this could be done just fine: just run multiple instances of workerd and load-balance across them. But doing this completely broke the model of Durable Objects, where each object is supposed to have a single instance globally, not one per workerd instance! This change fixes the problem, by introducing a new "cluster" mode. In cluster mode, you can run several workerd instances that are aware of each other and able to route requests to each other. All instances in a cluster run the same config. Mostly these workerd instances behave like they would normally, except when a request is made to a Durable Object, they coordinate to make sure only one workerd instance owns the DO, and others route to it. The design assumes a shared filesystem for underlying DO storage. All instances must be on the same filesystem. If all instances are on the same machine (useful for utilizing multiple cores), then this can be any local filesystem. Otherwise, it must be NFSv4, or some network filesystem that has exactly the same lock/lease semantics as NFSv4. The shared filesystem is also used for service discovery and locking. This is unconventional, but has a major advantage vs something like etcd or Consul: If a node loses its NFS lease, it simultaneously loses its locks *and* loses the ability to write to any open files. This provides "fencing": there's no way a node could continue writing after other nodes believe that it is dead. If locking were provided by a separate service, then it becomes extremely difficult to ensure that a node can't accidentally write after losing its lock due to a timeout. A complete design doc can be found here: https://gist.github.com/kentonv/cd8001237dc1181058193de0e7509972 The "Implementation Plan" in the design doc was largely written by AI (with several rounds of revision guided by me); the sections before that were largely written by me (with some AI review and minor changes). GPT 5.5, Opus 4.6, and Opus 4.7 were all employed. A blog post will come later. This particular commit was initially written by GPT 5.5 but heavily refactored by hand.
This test suite was written entirely by Claude Opus 4.7.
This flag was introduced by the persistent stubs feature, which was written and landed while DO clustering was still in flight. This commit is the first commit added after rebasing. It's essentially fixing a hidden merge conflict.
It's important that restoring a stub from storage does not immediately acquire the DO's lock or make network requests -- that should happen only if a request is made on the stub. Thus, we introduce a new `ClusterActorChannel` type (replacing `RpcActorChannel`) which does the lazily resolution. All stubs in cluster mode use this type, including when the actor ultimately ends up being local.
We need to drop the cluster lock on the DO: * No earlier than when we close the SQLite connection, since we need to stop all modifications after we've dropped the lock. (Practically speaking, SQLite has its own lock which prevents corruption, but if it is still held while the cluster lock is not, it'll cause confusion elsewhere.) * No later than when we remove the actor from the map, otherwise other nodes will keep routing to us and we might even get stuck in a loop of discovering we own the actor but it isn't in the map. However, historically, the actor could be removed from the map long before the sqlite connection was actually closed, since other references to the actor (e.g. stubs held by clients) could hold it open. The actor would block all storage operations in the meantime, but since it still held open the sqlite connection, the underlying files would still be locked. This was probably a bug even before the cluster changes. This commit changes things so that: * We forcefully close sqlite in `ActorSqlite::shutdown()`. The existing machinery for reset() (which closed and re-opened the database) makes this relatively easy to achieve. * We don't drop `ownershipLock` until after `shutdown()` has definitely been called.
Previously, after shutdown() was called, existing in-flight commits would continue running, potentially performing post-commit alarm updates. But at this point the actor could be dead and some other instance of the actor could have taken over, so this is incorrect. This commit makes us check for brokenness before these alarm updates. Probably we should really be cancelling the commit tasks, but that may have impact on SRS, so I have left it as a TODO for now.
The ResetListener machinery turned out to be less reusable than expected, since several implementations explicitly assume the reset is deleting all the DB content, not just closing. To fix this, we rename `beforeSqliteReset()` to `beforeSqliteClose()` and give it a parameter specifying if it's `close()` or `reset()`, and update all the implementations to do the right things. This is belt-and-suspenders (as Claude would say) with the previous commit. Before either commit, we could incorrectly cancel alarms after close() if an existing commit were in-flight, since after the close() attempts to read the metadata would incorrectly report that the alarm had been erased.
Specifically, this makes it so channel tokens that represent a directive to call `[restore]()` on some other token's target work when the first token is a remote actor in the cluster. The strategy we take is to figure out when the root of a restore chain turns out to be a remote actor and, in that case, send the full channel token (encoding the whole restore chain) to the remote machine, restoring it there, to handle any requests. This commit also changes the RPC interface exposed on the cluster port from WorkerDebugPort to a new WorkerClusterPort. The new interface deals strictly in (encrypted, MAC'd) channel token -- it does not accept arbitrary service props. This is a better fit for what we want and more secure to boot.
The RpcSystem needs to remain live all the way until process exit, but we were destroying it at drain time. We should only stop accepting new connections then. (Caught by Astra.)
…e routing loop. If a node opened a lockfile on disk and found it referenced itself, it would route to itself. But if no actor was actually loaded, the self-route would go load the lock again, and route to self again, in an infinite loop. This commit fixes this by adding a rule that when we encounter a self-reference, we don't optimistically route without checking whether the lock is held first. If the self-reference is stale, then it'll have no lock on it, so we'll end up creating a new lock and thus a new actor instance, avoiding the loop. Normally, skipping the lock is an important optimization, but in this case it doesn't matter because we always check for local presence in the actual in-memory actor table first. So, the only way we can end up finding a self-reference on disk is if it's actually stale or in certain narrow race conditions -- both unlikely.
We were already enforcing a timeout on the handshake after the connection was formed, but obviously the connect() itself can also time out.
In non-cluster mode, adding `abortAllActors()` to `handleDrain()` was arguably a regression, so make it apply in cluster mode only. (It is necessary in cluster mode since once we stop accepting new requests, any actor hosted locally would be unreachable by the cluster.) Additionally, make sure that we do not acquire a clustered actor after we've started draining.
This test takes 10s locally, and hit the 15s timeout in CI.
It turns out the production runtime sometimes calls ActorSqlite::shutdown() synchronously inside a VFS callback from sqlite (e.g. when hitting the storage size limit). close() there would abort the process. The edge runtime doesn't really need the `close()`, though -- it has its own way of doing things. So pull the close call up into server.c++, in all the places where we call shutdown().
This actually turned out pretty nice.
This proves to be too aggressive. In the edge runtime, we call getSqliteDatabase() post-broken in various cases (and we of course don't use the new close() function). This fixes broken tests in the internal build.
Cluster mode (cloudflare#6780) refused alarms: an AlarmScheduler per instance would fight over the namespace's metadata.sqlite. This makes them work when the instances share a local disk: - Every instance writes alarm times straight into the shared _cf_ALARM table (ClusterAlarmStore, used by ClusterActorSqliteHooks). The native VFS takes real file locks, so several processes can share the file. - One instance, the alarm leader, runs the AlarmScheduler. Leadership is an OFD lock on `alarm-leader` in the namespace directory, tried every second, so it moves to another instance when the leader goes away. - The leader picks up other instances' writes every 250 ms (AlarmScheduler::syncFromDb()): new rows are scheduled, moved times reschedule a waiting alarm or queue behind a running one, deleted rows cancel a waiting alarm. - After an alarm ran, the leader deletes its row only if it still holds the time that ran, so an alarm the handler set meanwhile on another instance survives (finishAlarm()). - A due alarm is delivered like any request (ClusterActorChannel): to the instance that holds the actor, or claimed by the leader when none does. The cluster's EventDispatcher now forwards runAlarm and abandonAlarm. - Instances starting together race to create the table and switch it to WAL; SQLite answers SQLITE_BUSY at once there, so initialization retries, and a leader that cannot start its scheduler gives the lock back. Tested with 3 processes on one machine: the time moved several times (runs once, at the latest), the handler throwing (retried with backoff), the holder killed with SIGKILL (claimed by the leader, runs once), the leader killed (another takes over, runs once). Not tested across machines (NFS). Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build>
The fork has no access to upstream's 16-core runners or remote cache, so this workflow builds the cluster-alarms branch from scratch on ubuntu-24.04 (within the 6-hour job limit) and uploads the binary. Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build>
Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build>
Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build>
Author
Review: #1, Durable Object alarms in cluster mode (head 74110df)Verdict: CHANGES REQUESTED The design holds up. Every process writes the shared Changes are still needed before this is used for real:
I reviewed by reading the code. I did not build or run anything (the botiverse-build run was still in progress). 1. Provenance
Notes on the workflow (Low):
2. Findings on 7ea530aH1. An existing integration test now fails, and there is no in-tree alarm test (High)
H2. Alarms are enabled in NFS mode, where a shared WAL database is unsafe (High for upstream use, Medium for this fork)
M1. A scheduler entry can be left in FINISHED with no task, and then that actor's alarms never fire on this leader (Medium)
M2. An alarm can be lost when the owner's "move later" write fails or is skipped (Medium)
M3. A draining leader keeps leadership and cannot claim actors (Medium)
M4. Synchronous shared-DB writes run on every process's event loop, with a 2 s busy wait (Medium, performance and robustness)
L1. Leadership is not given up when sync keeps failing (Low)
L2. Answer to Gogo's question (b), lock and scheduler startup (Low)
L3. Concurrent schema migration crashes all but one process (Low)
L4.
|
Author
|
Closed by the author, Gogo, through Stamp. Reason: Replaced by #2: same work plus the fixes from Imprint's review, based on botiverse/base-6780 (upstream 7a46b19, where cloudflare#6780 branched off) so it no longer conflicts with the fork's newer main.
|
stamp-build Bot
added a commit
that referenced
this pull request
Nov 3, 2026
* Update config capnp and worker-interface.capnp for clustering. This defines how the new feature is configured and exposed on the network. * Add OFD file lock primitive RAII wrapper around Linux OFD locks (fcntl F_OFD_SETLK) for per-fd file locking needed by the cluster DO ownership protocol. OFD locks are per open-file-description rather than per-process, which is essential since a single workerd process manages many DOs concurrently. Includes tryLock() (non-blocking), verifyHeld() (for NFSv4 lease verification), and move semantics. Non-Linux platforms fail at runtime rather than compile time so builds still succeed when cluster mode is unused. This code (and commit message) was entirely written by Opus 4.6. * Add cluster registry as a custom VatNetwork Implements ClusterRegistry, a capnp::VatNetwork for multi-node cluster RPC. Each node generates an X25519 keypair as its identity, binds a listener (Unix socket or ephemeral TCP port based on CIDR), and registers itself in a shared directory. Includes: - cluster.capnp: VatId and stub Level 3/4 types for VatNetwork params - X25519PublicKey: small wrapper with hex encoding and HashMap support - ConnectionImpl: per-peer Connection with MessageStream, idle timeout, and refcounted outbound caching - Handshake protocol: raw 32-byte public key exchange before RPC starts - Peer discovery: lazy directory scanning with mtime-based liveness - CIDR/IP mode: getifaddrs-based IP matching, OFD-locked registry file with periodic mtime heartbeat - Unix mode: socket-file-as-registry-entry, connect-based liveness The end-to-end RPC tests are present but currently failing; the non-network tests (keypair generation, self-bootstrap, peer discovery, self-connect) all pass. This code (and commit message) was originally written by Opus 4.6 but the code sucked and was heavily refactored by my human hand. * Add cluster DO ownership lock file protocol Implements ClusterLockManager, which manages the lock files that serialize ownership of Durable Objects across a workerd cluster. Each lock file is named after the actor ID and contains the owner's 32-byte X25519 public key (or is empty if unowned). acquireOrRoute() encapsulates the full protocol: - If the lock file is the expected 32-byte size, read the owner key and look it up in the ClusterRegistry. If the owner is live (or might be), return a WorkerdDebugPort capability via clusterRpc.bootstrap() — the RpcSystem short-circuits self-routing without going through the VatNetwork. - Otherwise (empty file or confirmed-dead owner), try to acquire an exclusive OFD lock. If granted, ftruncate, write our identity, fsync, and return an OwnedLock. If refused, retry with exponential backoff and jitter. No checksum is needed on the lock file payload: a single pwrite() of 32 bytes is atomic with respect to concurrent readers (both on local filesystems and over NFS, where it's a single RPC to the server). The writer's sequence (ftruncate → pwrite → fsync) means readers see either size 0 or the full 32-byte key, never a partial write. This is a deviation from the original spec, which called for a CRC32. OwnedLock is an RAII handle whose destructor truncates the file and releases the lock, returning the DO to the unowned pool. verifyStillOwned() is provided for the NFSv4 lease check that the SQLite VFS will call on every file open (Commit 4). This code (and commit message) was originally written by Opus 4.6 with a fair amount of manual cleanup. * Add a post-open callback to SQLite. This callback is invoked immediately after `open()` opens a file. In NFS mode we'll use this hook to verify the new open was opened on the same NFS lease as the registry lock. * Extend ChannelTokenHandler to use shared keys in cluster mode. In cluster mode, all members of the cluster need to use the same key for channel tokens, so they can exchange them. * Extend WorkerdDebugPort to include actor name with ID. If the actor ID was derived from a name, it's important to pass the name along with the ID so that `ctx.id.name` works correctly. * Make self-hosted Durable Objects scalable across a cluster. (This is the commit that wires everything up, but this commit message describes the overall series.) workerd has always been intended to be something you can run in production, in order to self-host Workers outside of Cloudflare. For many use cases it does in fact work for this. But, there has been one missing piece: Durable Objects. workerd has always supported running Durable Objects for the purpose of local testing, but only within the scope of a single instance running on a single thread. Any Durable Objects created would run within that same workerd instance. But in production, you probably want to utilize more than one thread on one machine. For stateless Workers, this could be done just fine: just run multiple instances of workerd and load-balance across them. But doing this completely broke the model of Durable Objects, where each object is supposed to have a single instance globally, not one per workerd instance! This change fixes the problem, by introducing a new "cluster" mode. In cluster mode, you can run several workerd instances that are aware of each other and able to route requests to each other. All instances in a cluster run the same config. Mostly these workerd instances behave like they would normally, except when a request is made to a Durable Object, they coordinate to make sure only one workerd instance owns the DO, and others route to it. The design assumes a shared filesystem for underlying DO storage. All instances must be on the same filesystem. If all instances are on the same machine (useful for utilizing multiple cores), then this can be any local filesystem. Otherwise, it must be NFSv4, or some network filesystem that has exactly the same lock/lease semantics as NFSv4. The shared filesystem is also used for service discovery and locking. This is unconventional, but has a major advantage vs something like etcd or Consul: If a node loses its NFS lease, it simultaneously loses its locks *and* loses the ability to write to any open files. This provides "fencing": there's no way a node could continue writing after other nodes believe that it is dead. If locking were provided by a separate service, then it becomes extremely difficult to ensure that a node can't accidentally write after losing its lock due to a timeout. A complete design doc can be found here: https://gist.github.com/kentonv/cd8001237dc1181058193de0e7509972 The "Implementation Plan" in the design doc was largely written by AI (with several rounds of revision guided by me); the sections before that were largely written by me (with some AI review and minor changes). GPT 5.5, Opus 4.6, and Opus 4.7 were all employed. A blog post will come later. This particular commit was initially written by GPT 5.5 but heavily refactored by hand. * Add integration tests for clustering mode. This test suite was written entirely by Claude Opus 4.7. * Drive-by: Fix compile_flags.txt for kj async library split. * Propogate the new `fromPersistentStub` flag correctly over RPC. This flag was introduced by the persistent stubs feature, which was written and landed while DO clustering was still in flight. This commit is the first commit added after rebasing. It's essentially fixing a hidden merge conflict. * Make cluster-routed actor channels resolve the actor lazily. It's important that restoring a stub from storage does not immediately acquire the DO's lock or make network requests -- that should happen only if a request is made on the stub. Thus, we introduce a new `ClusterActorChannel` type (replacing `RpcActorChannel`) which does the lazily resolution. All stubs in cluster mode use this type, including when the actor ultimately ends up being local. * Force closure of SQLite database before releasing DO lock. We need to drop the cluster lock on the DO: * No earlier than when we close the SQLite connection, since we need to stop all modifications after we've dropped the lock. (Practically speaking, SQLite has its own lock which prevents corruption, but if it is still held while the cluster lock is not, it'll cause confusion elsewhere.) * No later than when we remove the actor from the map, otherwise other nodes will keep routing to us and we might even get stuck in a loop of discovering we own the actor but it isn't in the map. However, historically, the actor could be removed from the map long before the sqlite connection was actually closed, since other references to the actor (e.g. stubs held by clients) could hold it open. The actor would block all storage operations in the meantime, but since it still held open the sqlite connection, the underlying files would still be locked. This was probably a bug even before the cluster changes. This commit changes things so that: * We forcefully close sqlite in `ActorSqlite::shutdown()`. The existing machinery for reset() (which closed and re-opened the database) makes this relatively easy to achieve. * We don't drop `ownershipLock` until after `shutdown()` has definitely been called. * Bugfix: Cease post-commit alarm updates when broken. Previously, after shutdown() was called, existing in-flight commits would continue running, potentially performing post-commit alarm updates. But at this point the actor could be dead and some other instance of the actor could have taken over, so this is incorrect. This commit makes us check for brokenness before these alarm updates. Probably we should really be cancelling the commit tasks, but that may have impact on SRS, so I have left it as a TODO for now. * Bugfix: ResetListeners need to distinguish reset() vs. close(). The ResetListener machinery turned out to be less reusable than expected, since several implementations explicitly assume the reset is deleting all the DB content, not just closing. To fix this, we rename `beforeSqliteReset()` to `beforeSqliteClose()` and give it a parameter specifying if it's `close()` or `reset()`, and update all the implementations to do the right things. This is belt-and-suspenders (as Claude would say) with the previous commit. Before either commit, we could incorrectly cancel alarms after close() if an existing commit were in-flight, since after the close() attempts to read the metadata would incorrectly report that the alarm had been erased. * Make persistent stubs work in clustered mode. Specifically, this makes it so channel tokens that represent a directive to call `[restore]()` on some other token's target work when the first token is a remote actor in the cluster. The strategy we take is to figure out when the root of a restore chain turns out to be a remote actor and, in that case, send the full channel token (encoding the whole restore chain) to the remote machine, restoring it there, to handle any requests. This commit also changes the RPC interface exposed on the cluster port from WorkerDebugPort to a new WorkerClusterPort. The new interface deals strictly in (encrypted, MAC'd) channel token -- it does not accept arbitrary service props. This is a better fit for what we want and more secure to boot. * Fix UAF of cluster RPC system. The RpcSystem needs to remain live all the way until process exit, but we were destroying it at drain time. We should only stop accepting new connections then. (Caught by Astra.) * Bugfix: A stale lockfile referencing this node could cause an infinite routing loop. If a node opened a lockfile on disk and found it referenced itself, it would route to itself. But if no actor was actually loaded, the self-route would go load the lock again, and route to self again, in an infinite loop. This commit fixes this by adding a rule that when we encounter a self-reference, we don't optimistically route without checking whether the lock is held first. If the self-reference is stale, then it'll have no lock on it, so we'll end up creating a new lock and thus a new actor instance, avoiding the loop. Normally, skipping the lock is an important optimization, but in this case it doesn't matter because we always check for local presence in the actual in-memory actor table first. So, the only way we can end up finding a self-reference on disk is if it's actually stale or in certain narrow race conditions -- both unlikely. * Bugfix: Enforce timeout on network connections. We were already enforcing a timeout on the handshake after the connection was formed, but obviously the connect() itself can also time out. * Bugfix: Only abort actors at drain time in cluster mode. In non-cluster mode, adding `abortAllActors()` to `handleDrain()` was arguably a regression, so make it apply in cluster mode only. (It is necessary in cluster mode since once we stop accepting new requests, any actor hosted locally would be unreachable by the cluster.) Additionally, make sure that we do not acquire a clustered actor after we've started draining. * Mark cluster-test-unix "large". This test takes 10s locally, and hit the 15s timeout in CI. * Bugfix: Pull db.close() out of ActorSqlite::shutdown(). It turns out the production runtime sometimes calls ActorSqlite::shutdown() synchronously inside a VFS callback from sqlite (e.g. when hitting the storage size limit). close() there would abort the process. The edge runtime doesn't really need the `close()`, though -- it has its own way of doing things. So pull the close call up into server.c++, in all the places where we call shutdown(). * Fix Justin's review nits. * Address Justin's nits, part 2: FixedArray<byte, 32> intsead of byte[32]. This actually turned out pretty nice. * Bugfix: Don't requineNotBroken() in getSqliteDatabase(). This proves to be too aggressive. In the edge runtime, we call getSqliteDatabase() post-broken in various cases (and we of course don't use the new close() function). This fixes broken tests in the internal build. * Support Durable Object alarms in cluster mode (local multi-process) Cluster mode (cloudflare#6780) refused alarms: an AlarmScheduler per instance would fight over the namespace's metadata.sqlite. This makes them work when the instances share a local disk: - Every instance writes alarm times straight into the shared _cf_ALARM table (ClusterAlarmStore, used by ClusterActorSqliteHooks). The native VFS takes real file locks, so several processes can share the file. - One instance, the alarm leader, runs the AlarmScheduler. Leadership is an OFD lock on `alarm-leader` in the namespace directory, tried every second, so it moves to another instance when the leader goes away. - The leader picks up other instances' writes every 250 ms (AlarmScheduler::syncFromDb()): new rows are scheduled, moved times reschedule a waiting alarm or queue behind a running one, deleted rows cancel a waiting alarm. - After an alarm ran, the leader deletes its row only if it still holds the time that ran, so an alarm the handler set meanwhile on another instance survives (finishAlarm()). - A due alarm is delivered like any request (ClusterActorChannel): to the instance that holds the actor, or claimed by the leader when none does. The cluster's EventDispatcher now forwards runAlarm and abandonAlarm. - Instances starting together race to create the table and switch it to WAL; SQLite answers SQLITE_BUSY at once there, so initialization retries, and a leader that cannot start its scheduler gives the lock back. Tested with 3 processes on one machine: the time moved several times (runs once, at the latest), the handler throwing (retried with backoff), the holder killed with SIGKILL (claimed by the leader, runs once), the leader killed (another takes over, runs once). Not tested across machines (NFS). Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> * ci(botiverse): build the Linux x64 binary on a standard runner The fork has no access to upstream's 16-core runners or remote cache, so this workflow builds the cluster-alarms branch from scratch on ubuntu-24.04 (within the 6-hour job limit) and uploads the binary. Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> * ci(botiverse): fix the workflow's YAML (a colon in a step name) Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> * ci(botiverse): unset the Android SDK variables after removing the SDK Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> * Cluster alarms: address review (#1) - Only clusters on one machine get alarms. Across machines (NFS mode, an IP `network`) a shared WAL-mode SQLite file isn't safe, so setAlarm() keeps failing there, as before, with a message that says why. - The alarm leader never writes the shared table. Each actor owns its row: after a successful run ActorSqlite clears its alarm, and an alarm set by the handler is written by the instance that holds the actor, both through ClusterActorSqliteHooks. Before, the leader deleted the row of the alarm that ran, which dropped a newer alarm if the holder's write had failed. A stale row is delivered again and ActorSqlite::armAlarmHandler() cancels it (and re-requests the time it does hold); alarms that just ran are not re-delivered for 5 seconds while the actor's own deletion is on its way. - When notifying an actor that its alarm was abandoned fails, the entry is forgotten instead of staying FINISHED with no task (which swallowed later times for that actor until the leader changed); the next sync delivers it again. - A draining instance gives up alarm leadership: it can no longer claim actors, so it could not deliver alarms either. - deleteAll() (resetting this instance's actors) no longer wipes the alarm table that every instance shares. Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> * Cluster test: alarms run once in the unix variant, still rejected in the CIDR (NFS) variant The test that expected alarms to be rejected in every cluster now runs only in the CIDR variant (NFS mode). The unix variant sets an alarm through one instance and checks, through another, that it ran exactly once, left no alarm behind and was not run again, and that the shared metadata.sqlite exists. Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> * test(botiverse): cluster-mode alarm cases (fresh cluster per case, exit code) Ten cases on three local processes, including the review's missing ones: setAlarm inside alarm(), the holder killed in the middle of the handler, the leader killed during a delivery, 300 alarms due at once, and 600 concurrent setAlarm calls across all processes (no SQLITE_BUSY). Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> * Cluster alarms: format (clang-format, Prettier); yield before dropping an entry after a failed abandon Formatting from tools/cross/format.py, which the lint check asked for. Also yields once before the abandon-failure path moves the task out of its own entry (review suggestion). Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> --------- Signed-off-by: Gogo <botiverse+gogo@agents.stamp.build> Co-authored-by: Kenton Varda <kenton@cloudflare.com> Co-authored-by: Gogo <botiverse+gogo@agents.stamp.build>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Durable Object alarms in workerd's cluster mode, for several workerd processes on one machine sharing a local disk, so a single host can use all its cores for Durable Objects (Raft Deck, #wg-deck).
This builds on cloudflare#6780 (kentonv's cluster mode, closed unmerged on 2026-09-11 because it is too slow over NFS; it works on a local filesystem). That PR refused alarms (
Durable Object alarms are not yet supported in cluster mode), and Deck depends on them, so it could not run Deck.The branch is cloudflare#6780's 25 commits, unchanged (re-committed so Stamp accepts them: author kept, tree identical to
922c2757), plus:Support Durable Object alarms in cluster mode (local multi-process):_cf_ALARMtable (ClusterAlarmStore/ClusterActorSqliteHooks). The native SQLite VFS takes real file locks, so processes can share it.AlarmScheduler. Leadership is an OFD lock onalarm-leader, retried every second, so it moves when the leader dies.AlarmScheduler::syncFromDb()). New rows are scheduled; a moved time reschedules a waiting alarm or queues behind a running one; a deleted row cancels a waiting alarm.finishAlarm()), so an alarm the handler set meanwhile on another instance survives.ClusterActorChannelto the instance that holds the actor, or the leader claims the actor if nobody does. The clusterEventDispatchernow forwardsrunAlarm/abandonAlarm; it used to throw "RPC connections don't yet support this event type".SQLITE_BUSYat once (busy_timeout doesn't cover it). Initialization retries, and a leader that can't start its scheduler gives the lock back.ci(botiverse): ...:botiverse-build.ymlbuilds the Linux x64 binary on a standardubuntu-24.04runner and uploads it. Upstream's builds need Cloudflare's 16-core runners and remote cache.How it was checked
workerd-alarm-test/cases.sh; the cases are Imprint's.SIGKILLed: the first delivery hits its stale socket; about 2 s later the leader claims the actor, and it runs once.SIGKILLed: another instance takes over within 1 s; the pending alarm runs once.9550dda, 4 processes on 4 cores, offices of 15, every user moving twice a second.chat_err rate), not dropped forwarding.Not done / risks