feat: scale Yjs across ECS tasks with durable document ownership

[JWThewes]

The current Yjs service keeps collaboration state in one task, so increasing concurrent editors can exhaust that task. Adding independent replicas would split live document state. This change adds opt-in document ownership and durable recovery so documents can be distributed across ECS tasks with manual or automatic scaling.

Closes #455.

Implementation

• Coordinate one owner per document using DynamoDB leases and fencing tokens, rendezvous routing, and authenticated private WebSocket forwarding. Checkpoint and release documents during rebalancing and shutdown.
• Serialize transactions per parent scope within each worker and retry explicitly transient transaction conflicts with bounded jitter. Preserve only the last confirmed lease deadline during transient failures; actual or unknown conditional failures still fail closed.
• Persist binary Yjs checkpoints in S3 with conditional DynamoDB manifests, and recover the committed version before synchronization. Explicit browser flushes wait for a durability receipt in cluster mode.
• Bound connections, loaded documents, document bytes, message sizes, and slow-consumer buffers. Correct awareness forwarding, remove periodic full-state synchronization, and add jittered reconnects.
• Serialize browser autosaves, trigger them only for local edits, and retain dirty changes across failures. Rotate artifact collaboration epochs when generated content is replaced so old snapshots and stale editors cannot restore an earlier artifact.
• Add guarded Terraform settings for task sizing, manual counts, and CPU/memory/capacity autoscaling, plus worker metrics, alarms, integration coverage, and an authenticated load workload. Explicitly disable ECS Availability Zone Rebalancing so existing services can adopt the required maximumPercent = 100 deployment configuration.
• Revoke document scopes when their intent/project is deleted. Binary snapshots are retained; parent deletion does not permanently purge them.

Validation

• Latest Node 24 Yjs integration suite after the contention fix: 144 tests / five files passed, including real WebSockets and DynamoDB Local. The earlier broader Yjs/intents/projects run passed 515 tests / 11 files on 3c18aadd.
• Frontend full suite: 550 tests / 79 files passed. Release/deployment tooling: 53 tests passed. Terraform validation and all five mocked deployment tests passed, including existing-service AZ rebalancing configuration.
• Formatting, linting, and secret checks passed. All eight PR checks are green on 25f62548. The unchanged root dependency tree has existing audit findings.
• Deployed dev verification (2026-09-11, us-east-1): existing ECS service upgraded in place; standalone browser collaboration/save/reopen passed; cluster mode recovered a checkpoint-only edit after graceful worker replacement with all browsers closed; manual scaling and browser edits preserved saved state.
• The first deployed cluster load smoke test exposed a transaction-contention bug, fixed in 25f62548. The same workload then converged with zero reconnects; a bounded AWS transaction probe completed 24/24 operations successfully after the fix.
• 100 clients for 180 seconds: both standalone and one-worker cluster passed 25-document and one-hot-document workloads at 1024 CPU units / 2048 MiB, each with 17,900 writes, full convergence, and zero reconnects. Cluster checkpoints contained every generated write. The smaller 256 / 512 dev worker passed 25 documents but saturated CPU and lost health with 100 clients on one hot document.
• 100 clients / 25 documents for 720 seconds during 1 → 2 → 4 → 2 scaling: all 100 clients converged and all 71,371 generated writes matched committed snapshots. This run recorded 420 reconnects, 137 rejected connection attempts, and three transient checkpoint conflicts. It passed convergence/durability with an availability caveat; it was not a zero-error scaling run.
• Both modes passed distinct-user concurrent editing, presence/departure, and wrong-subject scope-token rejection. Two browser tabs synchronized edits during load and shrink; final reopening on the restored worker recovered all markers.
• Final dev state: one healthy cluster worker at 256 / 512, all Yjs alarms OK, checkpoint tables retained. Temporary Cognito accounts and local credentials were deleted; two disposable-space membership entries await owner-session permission for cleanup.
• Abrupt owner death, injected AWS storage failures, and metric-driven autoscaling remain unverified in AWS. The load generator pauses writes while disconnected; received-update latency is not an availability measurement. Single-run results are not production capacity guarantees.
• Deployment used reviewed Yjs-scoped plans. Full-stack reconciliation remains outstanding because the broader plan includes additional Lambda updates and an old AgentCore image deletion referenced by runtime version 100.

Detailed procedure, measurements, failures, and cleanup status: #457 (comment)

Rollout and limitations

Cluster mode remains disabled by default. Production standalone sizing increases from 512 CPU units / 1024 MiB to 1024 / 2048; explicit overrides can retain the previous allocation. Other environments retain 256 / 512.

Enable cluster mode with exactly one worker first, verify persisted recovery, then increase the manual count. Exercise growth, shrinkage, owner failure, and storage errors in an isolated deployment before enabling automatic scaling. The single-worker transition briefly interrupts collaboration.

Horizontal scaling distributes documents; one hot document still has one owner. The two-second checkpoint interval is not a guaranteed recovery-point objective during storage failures. Snapshot retention and permanent deletion need an explicit policy before using cluster mode for data that must be erased with its parent.

Configuration, rollback steps, metrics, and the load workload are documented in docs/development/yjs-operations.md.

[JWThewes]

Deployment and runtime verification — 2026-09-11

Verified commit 3c18aadd582322b95d392cf23a0e60c7fe136074 in the existing dev / us-east-1 installation, using 256 CPU units / 512 MiB per worker. The ECS upgrade failure is fixed, and standalone and cluster browser checks passed. This is functional verification, not a load-capacity certification.

Existing-installation upgrade

The existing ECS service had Availability Zone Rebalancing ENABLED. ECS rejected the update to maximumPercent = 100 when rebalancing was omitted. The fix explicitly sets availability_zone_rebalancing = "DISABLED" in the same service update while retaining minimumHealthyPercent = 0 and maximumPercent = 100.

The existing service upgraded in place, first to the new standalone task and then to cluster mode with one worker. Both rollouts completed with healthy ALB targets and zero failed tasks. The frontend was deployed with scripts/deploy-frontend.sh dev and the browser reloaded.

Deployment scope: applies used reviewed, saved Terraform plans scoped to the Yjs module through scripts/deploy-terraform.sh dev --phase apply --plan-file … --skip-seed. A full-stack apply remains outstanding: the full plan also includes additional Lambda changes and deletion of a deposed AgentCore registry image whose digest is still referenced by runtime version 100. That cleanup was not applied. The live test establishes Yjs upgrade compatibility, not successful reconciliation of that broader plan.

Deployed checks

Check	Result and evidence
Standalone collaboration	PASS — two Chrome tabs exchanged edits in both directions without reload; presence and remote cursors appeared, and presence cleared when the second tab closed.
Standalone save/reopen	PASS — DynamoDB draft record contained the edits; after closing the last editor for more than 60 seconds, a fresh navigation recovered the content. No binary Yjs manifest was present in standalone mode.
Authentication rejection	PASS — WebSocket upgrade probes with missing or invalid Cognito credentials returned HTTP 401.
One-worker cluster persistence	PASS — decoded the exact S3 object version referenced by the DynamoDB manifest and verified its prompt content.
Cold recovery after worker replacement	PASS — procedure below excluded both browser-memory and REST-seed recovery.
Manual scaling 1 → 2 → 4 → 2 → 1	PASS — scaling bounds drove the actual ECS count. At two and four workers, healthy ALB targets and active membership matched the requested count. Editors remained open across the growth/shrink checks.
Ownership and forwarding	PASS — four test drafts occupied three owners at four workers; CloudWatch recorded proxied connections. Document ownership moved during membership changes.
Scale-in recovery	PASS — a draft whose owner drained temporarily became read-only, automatically reconnected without reload, preserved its text, and accepted a new saved edit. Edits before, during, and after shrinking reached the other shared-draft tab.
Final data integrity	PASS — all four draft records matched their decoded binary checkpoints after normalization of the editor's trailing newline; all remained DRAFT.

Cold recovery procedure: blocked REST requests only for the disposable draft after it loaded, inserted PR457-COLD-CHECKPOINT-ONLY, and confirmed that the committed binary checkpoint contained it while the business draft did not. Navigated the last editor to about:blank, gracefully replaced its ECS worker, removed the temporary blocking rule, and opened a fresh editor. The marker reappeared on a different owner from the same committed checkpoint version while the REST record still lacked it. A subsequent normal edit restored REST autosave and both representations matched. This was a graceful replacement, not an abrupt-kill test.

CloudWatch export covered 11:38:01–12:13:23 UTC, with 110 worker metric samples: summed PersistenceErrors = 0, summed RejectedConnections = 0, peak worker RSS 100.95 MiB, maximum reported event-loop p99 24.72 ms, and peak ProxiedConnections = 3. Both Yjs DynamoDB tables were ACTIVE with PAY_PER_REQUEST. These measurements describe the small browser workload only.

Scripts and checks

On Node 24:

• npm test -- --project yjs-server --project intents --project projects: 515 passed / 11 files, including DynamoDB Local and Gremlin integration.
• npm --prefix frontend test: 550 passed / 79 files.
• Release/deployment tooling suite: 53 passed. Fixed its stale Docker COPY assertion to validate the server's local runtime import closure.
• Terraform validation and mocked deployment tests: 5 passed; assertions cover explicit AZ rebalancing disablement in standalone, manual cluster, and automatic cluster configurations.
• Formatting, lint, and secret checks passed. All 8 GitHub PR checks are successful on this commit.

Remaining validation and final state

• The deployed load.js workload has not run: disposable test-user authorization is pending. Browser tabs used the same signed-in identity; distinct-user collaboration and sustained many-room/hot-document load are not claimed.
• Abrupt owner loss, injected cloud storage failures, and actual metric-driven autoscaling were not exercised in AWS. Local integration/mock coverage does not substitute for those deployment tests.
• Final deployment: cluster enabled, fixed count 1, one healthy target/member, no pending tasks, rollout completed. Local dev.tfvars now matches that state. Checkpoint storage is retained.
• Four disposable verification drafts remain in the test space for inspection. No agent workflow was started. Temporary browser request blocking was removed, and extra test tabs were closed.

[JWThewes]

Deployed load verification after 25f62548 — 2026-09-11

Standalone and one-worker cluster mode passed both 100-client workloads at 1024 CPU units / 2048 MiB. The live scaling run preserved every generated write, but it produced substantial reconnect activity and transient errors. The small development worker failed the concentrated hot-document workload; that failure is retained below.

Deployment and correctness fixes

• The existing-service ECS upgrade fix in 3c18aadd was applied and verified in place: set Availability Zone Rebalancing to DISABLED together with the required maximumPercent = 100, minimumHealthyPercent = 0. The earlier deployment-verification comment covers the initial upgrade, browser persistence, checkpoint-only cold recovery, and manual 1 → 2 → 4 → 2 → 1 checks.
• The first deployed cluster smoke test on the prior image failed: 10 clients, two documents, 20 seconds; 35 reconnects and no final convergence. A lease renewal treated a DynamoDB transaction conflict as lost ownership.
• A bounded AWS probe reproduced 11 transaction cancellations in 24 operations sharing the same parent-scope ConditionCheck. 25f62548 serializes each worker's transactions per parent scope, adds bounded jittered retries with the same idempotency token, and retains only the last confirmed lease deadline for explicitly transient failures. Actual or unknown conditional failures still fail closed.
• The fixed AWS probe completed 24/24 operations without an operation failure. The identical deployed smoke workload then passed with 190 writes, 10 connected clients, convergence, and zero reconnects.
• Latest local Node 24 Yjs integration suite: 144 tests / five files passed, including DynamoDB Local. All eight PR checks are green on 25f62548.

Steady workloads

Ran the unmodified lambda/yjs-server/load.js with Node 24, a dedicated temporary user, a disposable intent, 100 clients, 180 seconds, one update and awareness change per client per second, a 20 ms connection ramp, and no initial payload. All successful rows ended with all 100 clients connected and converged.

Mode	CPU units / MiB per worker	Clients / documents	Generated writes	Reconnects	Propagation p95 / p99, ms	Result
Standalone	256 / 512	100 / 25	17,900	0	236 / 359	Pass
Standalone	256 / 512	100 / 1	—	—	—	Aborted: worker health failure
Standalone	1024 / 2048	100 / 25	17,900	0	211 / 285	Pass
Cluster, one worker	1024 / 2048	100 / 25	17,900	0	167 / 231	Pass
Standalone	1024 / 2048	100 / 1	17,900	0	603 / 772	Pass
Cluster, one worker	1024 / 2048	100 / 1	17,900	0	674 / 1286	Pass

All successful steady workloads had zero persistence errors and zero rejected connections in their collected CloudWatch samples. In cluster mode, the committed S3 versions referenced by DynamoDB contained all 100 writers and exactly 17,900 writes for both workloads. Standalone runs created no binary Yjs manifests, as expected.

The 256 / 512 hot-document run saturated CPU at approximately 100%, reached approximately 23.4 seconds of event-loop delay, and lost ALB health. The load process was stopped; no final convergence or latency JSON was emitted. Memory remained low. The same workload completed in both modes at 1024 / 2048, with peak one-minute average CPU around 26%.

The larger standalone comparisons used a temporary task definition changing the cluster flag while retaining the same image, CPU, memory, networking, and existing checkpoint storage. The earlier normal standalone Terraform deployment was also verified. No saved cluster tables or snapshots were deleted to perform the mode comparison.

These are single-run measurements from one client host across the network, not production capacity guarantees. Hot-document latency samples are capped by the workload at the first 1,000,000 observations. One hot document still has one owner; adding workers distributes documents rather than splitting that document.

Continuous editing through manual scaling

Ran 100 clients / 25 documents for 720 seconds, changing worker count 1 → 2 → 4 → 2 through reviewed count-only Terraform plans and scripts/deploy-terraform.sh. Every worker kept 1024 CPU units / 2048 MiB and the same image.

• All requested worker counts reached healthy ALB targets and active membership. Live document ownership was 25 on one worker, then 13/12, then 8/7/6/4, then 14/11. Private forwarding was observed.
• Final result: 100 connected, converged, 71,371 generated writes. All 25 committed snapshots contained all 100 writers and exactly 71,371 writes.
• Observed 420 reconnects: 184 closes with code 1012, 58 with code 4401, and 178 with code 1006. The run crossed scope-token expiry and successfully reauthorized/reconnected. Client writes pause while disconnected, so this result is not uninterrupted availability.
• Across 54 collected worker metric samples, CloudWatch recorded three persistence errors and 137 rejected connection attempts. The checkpoint failures were transaction conflicts; final durable state recovered. This is a convergence/durability pass with an availability caveat, not a zero-error scaling run.
• Received-update propagation p95/p99 was 154/224 ms. That statistic does not represent time spent disconnected. Peak sampled event-loop p99 was about 45.7 ms; no queued output bytes were observed.

Browser and identity checks

Both modes passed checks with two distinct temporary Cognito identities: bidirectional presence, concurrent edits, presence removal on disconnect, and rejection (403) when one user's scope token was paired with the other user's JWT. Both users could access only the disposable test space and received 403 from the admin-user API.

On the fixed deployed image, two additional browser tabs shared a separate draft during the live load run. They recovered all earlier markers, synchronized new edits in both directions, displayed live presence, and accepted an edit during shrink without reloading. The latest markers matched both the business record and committed checkpoint; all four original fixtures remained drafts and matched their snapshots. No workflow was started.

After restoring development sizing and replacing the worker, a fresh browser navigation recovered all latest markers and connected successfully. The temporary browser account was then signed out, and both test tabs were closed.

Final state and remaining limits

Restored the saved dev configuration: cluster mode, one worker, 256 CPU units / 512 MiB, fixed image 21f3b79b0d27db31, ECS task definition revision 27. ECS desired/running = 1/1, pending = 0, deployment completed, one healthy ALB target/member, Availability Zone Rebalancing disabled, and all four Yjs alarms OK. Both DynamoDB tables are ACTIVE. The two temporary standalone task definitions are inactive; existing tables and checkpoints were retained.

Both temporary Cognito accounts were signed out globally and deleted, with deletion verified. The private credential directory was removed. Two membership entries remain in the disposable test space: removing those entries requires permission to use the existing owner browser session, which is pending. Test drafts were retained for inspection.

Abrupt owner death, injected AWS storage outages, and metric-driven autoscaling were not exercised in AWS. Graceful replacement, manual scaling, and the local fault coverage should not be read as proof of those scenarios.

Deployment used Yjs-scoped recovery plans. Full-stack reconciliation remains outstanding: the broader plan includes unrelated Lambda updates and deletion of an older AgentCore image still referenced by runtime version 100. That cleanup was not applied. No merge was performed.