diff --git a/.github/audit/hosted.md b/.github/audit/hosted.md index 1a4f2e31e..8bb58fdf9 100644 --- a/.github/audit/hosted.md +++ b/.github/audit/hosted.md @@ -17,8 +17,8 @@ unreachable, report those two checks as `UNVERIFIABLE`. Read `docs/specs/hosted.md`, `docs/specs/one-time.md` (its "Wire contract", "Hosted rendezvous", and "Phone page"), `docs/specs/relay.md` (its "HTTP API", -"Setup tokens and the pairing QR", and "WebAuthn without a WebAuthn library", -whose semantics the Hosted Relay keeps), `hosted/server/`, `hosted/src/`, +"Setup tokens and the pairing QR", "WebAuthn without a WebAuthn library", and +"Routing", whose semantics the Hosted Relay keeps), `hosted/server/`, `hosted/src/`, `hosted/scripts/`, `hosted/wrangler.jsonc`, `hosted/wrangler.relay.jsonc`, `hosted/wrangler.voice.jsonc`, `remote-lib-common/src/remote/one-time-wire.ts`, @@ -130,6 +130,20 @@ Be adversarial, and go past the `FAIL IF` list. Ask specifically: Relay must read no cookie, and the rendezvous authorizes nothing. Pocket and `/connect/` share the relay origin; check what each page's policy lets it reach of the other. +- **Can one account's relay socket reach another's?** Trace an upgrade + through `relaySocketRoutes` (`hosted/server/relay-sockets.ts`) into + `RelayRoom` (`hosted/server/relay-room.ts`): the object must be named only + from the account a token resolved to, refuse any request or RPC naming + another, and receive no header or token of the caller's; a web page must not + open a Burrow socket, nor another origin a Client socket. Look for routing + state kept in memory that a hibernated object would lose, a socket torn down + twice or routed after its close began, a frame parsed before its length is + bounded or bounded in characters rather than bytes, a `ct` read, decoded, + logged, or stored outside the shared frame layer's field copy, a Client cap + one socket can evict past, a session that outlives its alarm, a ping that + wakes the object, a Burrow socket accepted on a row removed after the token + check, and a removed or de-entitled Burrow whose socket outlives the hourly + sweep. - **Can the rendezvous become more than a handshake pipe?** Trace a frame through `OneTimeRoom`: nothing may read, keep, or log it, and the length, type, and count bounds must close both ends before a byte past them is diff --git a/AGENTS.md b/AGENTS.md index 218c82b7d..f010f9d44 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -122,7 +122,7 @@ Six sibling lints run in `pnpm test`. Five enforce one invariant a spec states i | `scripts/loopback-lint.mjs` (`pnpm lint:loopback`) | `docs/specs/security-local.md` -> "Loopback Listeners": a loopback bind is not an access control — a new listener references a guard module or is allowlisted with a reason. | | `scripts/deploy-lint.mjs` (`pnpm lint:deploy`) | `docs/specs/security-remote.md` -> "Credentials at rest" and "Network posture (self-hosted)": the installer controls binding all three of `deploy/local/install-{macos,windows,linux}`. | | `scripts/ps1-cmdlet-lint.mjs` (`pnpm lint:deploy`) | Every `Verb-Noun` call in `deploy/local/install-windows.ps1` uses an approved verb and a noun that is not this project's vocabulary. No job has a PowerShell, so this is the Windows installer's only syntax gate. | -| `scripts/e2e-lint.mjs` (`pnpm lint:e2e`) | The structural half of `docs/specs/security-remote.md` -> "Remote Control": one Noise suite with no selector, no JavaScript curve, no legacy relay discriminant, no Relay-side protocol-v1 type, no one-time reader in the Relay or `BurrowRuntime`, no store in the one-time phone, no checked-in service worker, no optional field on a ciphertext or transcript; and `docs/specs/security-hosted.md` -> "Rendezvous boundary": no frame read in Hosted's one-time room. | +| `scripts/e2e-lint.mjs` (`pnpm lint:e2e`) | The structural half of `docs/specs/security-remote.md` -> "Remote Control": one Noise suite with no selector, no JavaScript curve, no legacy relay discriminant, no Relay-side protocol-v1 type, no one-time reader in the Relay or `BurrowRuntime`, no store in the one-time phone, no checked-in service worker, no optional field on a ciphertext or transcript; and `docs/specs/security-hosted.md`: no frame read in Hosted's one-time room, or decoded or kept in its `RelayRoom`. | `scripts/spec-lint-selftest.mjs` plants one defect per finding check in the spec lint. The `deploy`, `e2e`, and `loopback` lints carry self-tests that mutate each rule in whichever direction it points: a present-control rule has its control deleted (and, for exact-count rules, a copy added), a `forbidden` rule has the banned text appended. `scripts/e2e-lint-selftest.mjs` is mostly the second kind; `scripts/deploy-lint-selftest.mjs` mostly the first. Either way the lint must go red. **A rule added to one of these lints without its self-test case is not enforced** — it is a claim that something is checked. They share plumbing, and only that, through `scripts/lint-kit.mjs`. `scripts/installer-verify-test.mjs` (also `pnpm lint:deploy`) runs the installer shell helpers lint can only read, extracted from the shipped files; `scripts/ps1-cmdlet-lint-selftest.mjs` carries the `ps1-cmdlet` lint's mutations. `pnpm test` also runs `scripts/clamp-issue-body-selftest.mjs`, the test for `scripts/clamp-issue-body.mjs` (the helper the audit workflows use to keep an issue body postable); it lives at the repo root because its callers do. diff --git a/docs/specs/hosted.md b/docs/specs/hosted.md index 4148470ae..e90f758c8 100644 --- a/docs/specs/hosted.md +++ b/docs/specs/hosted.md @@ -1,7 +1,7 @@ # Dormouse Hosted accounts > See `docs/specs/glossary.md` for Burrow, Client, Relay, and Session vocabulary. -> Owns the Hosted account application, the Hosted Relay's account-scoped routes, and the deployment of Hosted's three Workers. The one-time rendezvous the relay Worker serves belongs to `docs/specs/one-time.md` -> "Hosted rendezvous"; the Relay's shared route semantics to `docs/specs/relay.md` -> "HTTP API"; remote authorization to `docs/specs/remote-security-model.md`. +> Owns the Hosted account application, the Hosted Relay's account-scoped routes and sockets, and the deployment of Hosted's three Workers. The one-time rendezvous the relay Worker serves belongs to `docs/specs/one-time.md` -> "Hosted rendezvous"; the Relay's shared route and routing semantics to `docs/specs/relay.md` -> "HTTP API" and "Routing"; remote authorization to `docs/specs/remote-security-model.md`. ## Application boundary @@ -9,8 +9,8 @@ | Worker | Origin | Serves | Holds | |---|---|---|---| -| `dormouse-hosted` | `https://hosted.dormouse.sh` | the account frontend, `/api/auth/*`, `/api/providers`, `/api/ready`, voice tokens, the Relay's account routes ("Burrow enrollment") | the login cookie, auth secrets, Hyperdrive, the approval rate limit | -| `dormouse-relay` | `https://relay.dormouse.sh` | the Hosted Relay and Pocket ("Relay"), the one-time rendezvous and `/connect/` (`docs/specs/one-time.md` -> "Hosted rendezvous") | Hyperdrive, `OneTimeRoom`, the one-time, sign-in, setup, and enrollment rate limits, `ACCOUNT_ORIGIN`, `RELAY_ENROLL_SECRET` | +| `dormouse-hosted` | `https://hosted.dormouse.sh` | the account frontend, `/api/auth/*`, `/api/providers`, `/api/ready`, voice tokens, the Relay's account routes ("Burrow enrollment") | the login cookie, auth secrets, Hyperdrive, the approval rate limit, a binding to the relay's `RelayRoom` | +| `dormouse-relay` | `https://relay.dormouse.sh` | the Hosted Relay, its sockets, and Pocket ("Relay", "Relay sockets"), the one-time rendezvous and `/connect/` (`docs/specs/one-time.md` -> "Hosted rendezvous") | Hyperdrive, `OneTimeRoom`, `RelayRoom`, the one-time, sign-in, setup, and enrollment rate limits, `ACCOUNT_ORIGIN`, `RELAY_ENROLL_SECRET` | | `dormouse-voice` | `https://voice.dormouse.sh` | speak and the history sweep ("Managed voice") | `ELEVENLABS_API_KEY`, Hyperdrive | Every Worker answers `/api/health`, 404s anything else under its non-page prefixes (`/api`, and the relay's `/ws` too), `/dev/*`, or `/__test/*`, and answers a thrown request 503 under `secureHeaders`. The account falls back to its SPA assets, the relay to Pocket's. The 421 origin gate, each Worker's bindings mapper, and the cookie routes' exact-`Origin` check are `docs/specs/security-hosted.md` -> "Origin boundary" (rationale). @@ -100,16 +100,16 @@ The relay Worker serves the self-host Relay's HTTP API to many accounts: the pat | `POST /api/setup/retire` | Spends only a token one of the session's account's Burrows minted | | `POST /api/signin/finish` | The asserted credential's account; `accountId` is its user ID. 401 `NOT_ENTITLED_ERROR`, and no session, for an account not entitled | | `POST /api/reauth/begin`, `/finish` | Only the session's account's credentials and nonces | -| `GET /api/burrows` | The session's account's Burrows, each `online: false`: no relay socket reaches Hosted yet | +| `GET /api/burrows` | The session's account's Burrows, each `online` while it holds a live socket in the account's `RelayRoom` ("Relay sockets") | | `POST /api/burrow/setup-token` | 401 for a removed Burrow, 403 `NOT_ENTITLED_ERROR` for an owner not entitled | | `POST /api/burrow/enroll` | Always 401 `UNAUTHORIZED_ERROR`: Hosted has no setup password; a Burrow enrolls by device code ("Burrow enrollment") | -| `GET /api/push/config` | `{ applicationServerKey: null }`: push is off; every other push route, `/ws/*`, and `GET /api/hello` (the self-host installers' probe) are 404 | +| `GET /api/push/config` | `{ applicationServerKey: null }`: push is off; every other push route and `GET /api/hello` (the self-host installers' probe) are 404 | | `GET /*` | Pocket (below) | A session-gated route answers a session of an account no longer entitled with the expired session's 401 `UNAUTHORIZED_ERROR`, so Pocket returns to sign-in. - **Must keep the Relay's state in Postgres** (`hosted/server/dormouse-migrations/002_relay.sql`). A sign-in challenge is the challenge row with no Burrow. -- **Must resolve each bearer and its owner's entitlement in one query joining `"user"`** (`sessionByToken`, `burrowByToken`). The entitlement is `isAdmin` ("Managed voice"). Reserved: the relay socket upgrades (Future item 4) call the same lookups with the query-parameter token. +- **Must resolve each bearer and its owner's entitlement in one query joining `"user"`** (`sessionByToken`, `burrowByToken`), the socket upgrades' query-parameter token included. The entitlement is `isAdmin` ("Managed voice"). - **Must cap every table a caller grows, keyed by whoever grows it**, as `docs/specs/relay.md` -> "Guardrails" does: setup tokens and setup challenges per Burrow (`MAX_TOKENS_PER_BURROW`), presence nonces per session (`MAX_PENDING_REAUTH_NONCES_PER_SESSION`), and sessions per account (`MAX_SESSIONS_PER_ACCOUNT`), each evicting its key's own oldest; passkeys per account are refused at the cap (rationale). A capped insert runs under its key's advisory lock and, in one statement, prunes that key's own expired rows, trims its live rows, and inserts; it never touches another key's rows. - **Must sweep every Relay table's expired rows from the relay's hourly Cron Trigger** (rationale); sign-in challenges, minted unauthenticated and flat, are bounded by it and the per-address limit. - **Must answer 429 with `Retry-After` past the per-address limit on `signin/*` (`RELAY_SIGNIN_LIMIT`) and `setup/begin`/`finish` (`RELAY_SETUP_LIMIT`)**, before the body limit and any database read: 30 a minute, a ceremony's two routes sharing one budget (rationale). @@ -119,6 +119,27 @@ A session-gated route answers a session of an account no longer entitled with th Source of truth: `relayApiRoutes` / `sweepExpired` in `hosted/server/relay-api.ts`; `sessionByToken` / `burrowByToken` / `requireSession` / `requireBurrow` in `hosted/server/relay-auth.ts`; `hosted/server/dormouse-migrations/002_relay.sql`; `triggers` and `ratelimits` in `hosted/wrangler.relay.jsonc`; `pocketRoutes` in `hosted/server/pocket.ts`; `relayRules` / `relayPathKind` in `hosted/server/headers.ts`; `stageRelay` in `hosted/scripts/stage-relay.mjs`; `checkRegistration` / `verifySigninAssertion` in `remote-lib-common/src/remote/relay-common.ts`. Pinned by `hosted/server/tests/relay.test.ts`, `hosted/server/tests/pocket.test.ts`, `hosted/server/tests/boundary.test.ts`, `hosted/scripts/stage-relay.test.mjs`, and `remote-lib-common/test/relay-common.test.mjs`. +## Relay sockets + +The relay Worker serves `GET /ws/burrow` and `GET /ws/client` at the self-host paths, token in `WS_TOKEN_PARAM`, and routes them by `docs/specs/relay.md` -> "Routing", every rule, through the same frame layer and bounds. Only the differences; `docs/specs/security-hosted.md` -> "Relay boundary" holds the checks. + +| Route | Admitted | Refused | +|---|---|---| +| `GET /ws/burrow` | no `Origin`, a Burrow token whose owner is entitled | 403 `forbidden` with any `Origin`; 401 `UNKNOWN_BURROW_TOKEN_ERROR`; 403 `NOT_ENTITLED_ERROR` | +| `GET /ws/client` | `Origin` exactly `APP_ORIGIN`, a live session of an entitled account | 403 `forbidden`; 401 `UNAUTHORIZED_ERROR` for any session refused, which Pocket's probe of `GET /api/burrows` reads as expiry | + +- **Must route each account's sockets through one `RelayRoom` Durable Object, named `idFromName(userId)` from the account the token resolved to**, holding all its Burrow and Client sockets. Another account's Burrow is in another object, so a cross-account binding is impossible, not refused. +- **Must hand the object a fresh request**: the upgrade, the account, and the Burrow's id or the session's `expiresAt` as search parameters (`RELAY_ROOM_PARAMS`); never a header, token, or cookie of the caller's. +- **Must store only the account id**, written at the object's first upgrade; a request or RPC naming another account is refused and logged with no frame content. +- **Must recheck a Burrow's row in the object before accepting its socket**, under `blockConcurrencyWhile`: enrolled, the account's, its owner entitled, else the Worker's 401 or 403. A removal committed after the Worker's token check is refused here; one committed later has its `closeBurrow` delivered after the accept. **Never open a database connection in the object** (rationale): it reads rows through the relay Worker's `RelayRows` entrypoint, reached by `ctx.exports`. **Must bound every row read at `RELAY_ROW_READ_TIMEOUT_MS`**, under the runtime's 30 s `blockConcurrencyWhile` reset: past it the upgrade answers 503 and the sweep leaves its Burrows to the next. +- **Must accept every socket through the Hibernation API**, tagged by role and id, keeping the routing state in its attachment — role, `clientId`, bound `burrowId`, `expiresAt` — so a woken object rebuilds it from `ctx.getWebSockets`. +- **Must measure a text frame in UTF-8 bytes against `MAX_RELAY_FRAME_BYTES` before parsing it**, as the self-host `maxPayload` counts, closing 1009 past it; a binary frame is dropped. The Client cap is per account object. +- **Must keep one alarm at the earlier of the earliest held Client's `expiresAt` and, while a Burrow socket is held, the next sweep**, at most `RELAY_ROOM_SWEEP_MS` (an hour) away. Each alarm closes every expired Client 1008 `unauthorized`, its Burrow told `client-gone`; then reads every held Burrow's row in one query and closes 4001 each removed, another account's, or de-entitled, its Clients told `burrow-gone`; then re-arms, or clears while nothing is held. An upgrade only brings the alarm earlier and a close leaves it. It replaces the self-host expiry and revocation sweeps. +- **Must answer `RELAY_PING` with `RELAY_PONG` as the object's auto-response**, which never wakes it, reaches no handler, and is never forwarded. **A socket whose last auto-response is older than `RELAY_IDLE_TIMEOUT_MS` is gone**, retired with 1001; a socket that never pinged is never judged. It replaces the self-host heartbeat. Two gaps are accepted: a Burrow that never pings is never judged, since every build that can enroll with Hosted pings; and at the Client cap a backgrounded Pocket, its pings paused, counts as gone once silent that long — the cap is far above real use, and the Burrow reaps its session at 120 s anyway. +- **Must close a removed Burrow's socket from the account Worker's `DELETE /api/relay/burrows/:burrowId`**, through its `RELAY_ROOM` binding (`script_name` the relay) calling `closeBurrow`: 4001, its Clients `burrow-gone`. **Must answer 204 once the row is gone, even if the close fails**, logging it; the sweep closes the socket. No route reaches either RPC; `GET /api/burrows` reads `online` through `onlineBurrows`, beside its query. + +Source of truth: `relaySocketRoutes` in `hosted/server/relay-sockets.ts`; `relayRoom` / `RELAY_ROOM_PARAMS` / `RELAY_ROOM_SWEEP_MS` / `RELAY_ROW_READ_TIMEOUT_MS` in `hosted/server/relay-room-contract.ts`; `RelayRoom` in `hosted/server/relay-room.ts`; `RelayRows` in `hosted/server/relay-rows.ts`; `forwardUpgrade` / `refuseSocket` in `hosted/server/socket-room.ts`; `relayAccountRoutes` in `hosted/server/relay-account.ts`; `exceedsRelayFrameBytes` in `remote-lib-common/src/remote/relay-routing.ts`; `durable_objects` and `migrations` in `hosted/wrangler.relay.jsonc` and `hosted/wrangler.jsonc`. Pinned by `hosted/server/tests/relay-room.test.ts`, which also runs the routing cases every Relay passes (`remote-lib-common/test/harness/relay-parity.mjs`), and `hosted/server/tests/workers.test.ts`. + ## Burrow enrollment A Burrow joins an account by device code, in place of the self-host setup password. The account owns the Burrow; the Burrow's own ACL still authorizes every Client. The relay serves the Burrow's two routes, the account the rest; wire types are `BurrowEnrollBeginResponse` / `BurrowEnrollPollResponse` in `remote-lib-common/src/remote/wire.ts`. @@ -144,7 +165,7 @@ Errors are the managed-voice cookie routes' ("Managed voice"), except that 403 f - **Must refuse approval from a login older than `LOGIN_FRESH_AGE_MS`**, 403 `RECENT_LOGIN_REQUIRED`, reading `get-session`'s `createdAt` in that route alone and failing closed when it is missing or unparsable; the gate the voice routes share never reads it. - **Must count every approval attempt** against `RELAY_APPROVE_LIMIT` (10 a minute per account) before reading the body: 429 with `Retry-After` past it. - **Must answer a malformed code 400**, forgiving case, spaces, and dashes, and a second approval of a live code 409 `ALREADY_APPROVED`, whichever account sends it: a live approval never moves. An expired one is replaced. -- **Must remove by deleting the Burrow's row**, its setup tokens and setup challenges cascading, so `burrowByToken` finds nothing on any relay route; its live socket ends with the sockets (Future item 4). +- **Must remove by deleting the Burrow's row**, its setup tokens and setup challenges cascading, so `burrowByToken` finds nothing on any relay route, **then close its live socket** ("Relay sockets"). - **Must take the `/enroll` fragment before render and erase it from history**, as `/connect/` does, and hold the code in memory only: through email sign-in and back, never into storage. A fragment change on `/enroll` takes the new code the same way without reloading; elsewhere it does nothing. Provider sign-in leaves the page, so the user opens the link again. The page shows the code in the user-code role with "Approve only if Dormouse on your computer is showing this code right now." and offers Approve only within the recent-login window, Sign in again otherwise. The account page's Computers section lists each Burrow by its ID's first eight characters and enrollment date (the Relay keeps no name), with Remove. @@ -155,7 +176,7 @@ Source of truth: `relayApiRoutes` in `hosted/server/relay-api.ts`; `relayAccount **Must run local development with `dor tool hosted` inside Dormouse.** A single `http://localhost:` origin, bound to loopback on an OS-assigned port unless `PORT` pins one, serves Vite and Node auth, with a disposable development database, and the voice token and Relay account routes but never speak, which every Hosted build reaches only at `https://voice.dormouse.sh`. Host, Origin, and Fetch Metadata checks guard the local captured-email inbox; no production entry imports an inbox or test-control handler. `dor tool one-time` runs the relay Worker on loopback without a database, so its Relay routes answer 503 (`docs/specs/one-time.md` -> "Dev loop"). -**Must verify the three production Worker bundles and run the consumer's integration suite before release.** Root `pnpm test` runs the `hosted/scripts/*.test.mjs` deploy suites and `test:miniflare`, the Docker-free Miniflare suites (the rendezvous, Pocket's serving, and the three Workers' boundary); the rest of `pnpm test:hosted`'s vitest half runs only there, `workers.test.ts` and `relay.test.ts` needing Docker. The test entry alone injects the packed Better Auth deterministic module. Simulated callbacks do not certify provider registrations; production acceptance requires real browser login with each enabled provider and email delivery. +**Must verify the three production Worker bundles and run the consumer's integration suite before release.** Root `pnpm test` runs the `hosted/scripts/*.test.mjs` deploy suites and `test:miniflare`, the Docker-free Miniflare suites (the rendezvous, Pocket's serving, and the three Workers' boundary); the rest of `pnpm test:hosted`'s vitest half runs only there, `workers.test.ts`, `relay.test.ts`, and `relay-room.test.ts` needing Docker. The test entry alone injects the packed Better Auth deterministic module. Simulated callbacks do not certify provider registrations; production acceptance requires real browser login with each enabled provider and email delivery. **Must keep production, test, and preview databases and credentials separate.** The development and preview entries are email-only. Production configuration and operator steps live in `hosted/README.md`. @@ -165,7 +186,7 @@ Source of truth: `allowedDevRequest` in `hosted/server/dev-host-guard.ts`; `host **Must deploy only verified same-repository PR merge revisions touching Hosted or its shared build inputs.** Drafts qualify; forks receive no deployment credentials. Changed paths include rename sources and all API pages. Deployment runs serialize per PR without cancellation; close/merge cleanup ignores path filtering and tolerates absent resources. -**Must isolate each PR in three persistent workers.dev Workers (`dormouse-{hosted,relay,voice}-pr-N`), one uncached Hyperdrive all three share, and a Neon branch from an empty dedicated preview project**, all reused until close. Preview configs exclude production routes, triggers, and credentials; runtime bindings cannot enable OAuth, Postmark, or ElevenLabs. **Must give each relay preview its own Durable Object namespace, each preview preview-only rate-limit namespaces, and the relay preview an `ACCOUNT_ORIGIN` naming its account preview**, and delete every preview Worker with `force`. A preview's one secret (the account's `AUTH_SECRET`, the relay's `RELAY_ENROLL_SECRET`) is the HMAC of `PREVIEW_AUTH_SECRET` with its Worker's name (`previewSecrets`). The smoke checks what production's does ("Production releases"), with the captured-mail login in place of providers, retrying each part on its own. +**Must isolate each PR in three persistent workers.dev Workers (`dormouse-{hosted,relay,voice}-pr-N`), one uncached Hyperdrive all three share, and a Neon branch from an empty dedicated preview project**, all reused until close. Preview configs exclude production routes, triggers, and credentials; runtime bindings cannot enable OAuth, Postmark, or ElevenLabs. **Must give each relay preview its own Durable Object namespaces, the account preview's `RELAY_ROOM` binding its relay preview's, each preview preview-only rate-limit namespaces, and the relay preview an `ACCOUNT_ORIGIN` naming its account preview**, and delete every preview Worker with `force`. A preview's one secret (the account's `AUTH_SECRET`, the relay's `RELAY_ENROLL_SECRET`) is the HMAC of `PREVIEW_AUTH_SECRET` with its Worker's name (`previewSecrets`). The smoke checks what production's does ("Production releases"), with the captured-mail login in place of providers, retrying each part on its own. **Must run cleanup from the base branch's checkout, never the closed PR's.** @@ -177,7 +198,7 @@ Source of truth: `touchesHosted` in `hosted/scripts/changed.mjs`; `.github/workf **Must deploy only manually selected main revisions after Hosted tests/build and accepted clean package provenance.** `verifyPackages` checks both installed packages' clean, matching provenance; `productionConfig` holds each config to its Worker's pinned name and origin and lone custom domain, and the relay's `ACCOUNT_ORIGIN` to the account's origin; preflight checks uncached Hyperdrive, matching migration/runtime database identity with distinct roles, and each Worker's own secret names (the voice's is `ELEVENLABS_API_KEY`, the relay's `RELAY_ENROLL_SECRET`). Back up, encrypt, decrypt, and restore-test before applying migrations; upload only the encrypted archive. Deploy relay, voice, then account, stopping at a failure; the relay must pass its revision check and `oneTimeSmoke` before the next deploy (rationale). Production has no public candidate URL. -**Must only append Durable Object migrations**: a deployed tag is never edited or removed, and Cloudflare refuses a rollback across one, so each is a rollback floor. The account keeps the `v1` that created `OneTimeRoom` and appends `v2` deleting it; the relay starts its own `v1`. A deploy restarts every room, dropping links still waiting or mid-handshake; a session already on its direct path never touches Hosted. Live verification checks each Worker's revision and runs `oneTimeSmoke` on the relay once the relay's revision check passes, whatever the account's outcome; a failed smoke reports every failed part. +**Must only append Durable Object migrations**: a deployed tag is never edited or removed, and Cloudflare refuses a rollback across one, so each is a rollback floor. The account keeps the `v1` that created `OneTimeRoom` and appends `v2` deleting it; the relay has its own `v1` (`OneTimeRoom`) and `v2` (`RelayRoom`). A deploy restarts every room, dropping links still waiting or mid-handshake, and every relay socket, which its end reconnects; a session already on its direct path never touches Hosted. Live verification checks each Worker's revision and runs `oneTimeSmoke` on the relay once the relay's revision check passes, whatever the account's outcome; a failed smoke reports every failed part. **Must bound retries.** Health GETs require the selected revision, sharing six five-second retries for transport failures or healthy stale revisions. Retry rate-limited OAuth once; never replay POSTs after transport failures. Production repeats only the relay and voice smokes, up to six times 10 s apart (rationale). @@ -190,4 +211,4 @@ Source of truth: `.github/workflows/hosted-production.yml`; `productionConfig` / 1. Deploy the configured providers and pass real production acceptance. pgstencil includes the Microsoft fix; personal and work/school callbacks need acceptance. 2. Add per-browser login listing/revocation, sign-out-everywhere, and account recovery before broad paid use. Revisit the fixed 24-hour login lifetime for daily voice use. 3. Managed voice beyond the admin slice: a real entitlement or licence replacing `ADMIN_EMAIL`, credentials scoped for non-admin accounts, per-account quotas, usage accounting, and spending bounds beyond the fixed daily cap, and explicit text/redaction disclosure. -4. Hosted Relay beyond "Relay" and "Burrow enrollment": relay sockets and `online`, live sockets ending on removal, desktop enrollment, and push — **saas-multitenant** in `docs/specs/relay.md` and **remote-network** in `docs/specs/remote-network.md`. Account login never replaces Burrow pairing and authorization. Paid security claims require independent review. +4. Hosted Relay beyond "Relay", "Relay sockets", and "Burrow enrollment": desktop enrollment and push — **saas-multitenant** in `docs/specs/relay.md` and **remote-network** in `docs/specs/remote-network.md`. Account login never replaces Burrow pairing and authorization. Paid security claims require independent review. diff --git a/docs/specs/hosted.rationale.md b/docs/specs/hosted.rationale.md index 47faac9c6..3ba087188 100644 --- a/docs/specs/hosted.rationale.md +++ b/docs/specs/hosted.rationale.md @@ -40,6 +40,10 @@ Sweep interval (2026-09-30): hourly, not the voice sweep's five minutes. Each pa Rate limits (2026-09-30): `signin/*` and `setup/begin`/`finish` are the unauthenticated routes that reach Postgres. A ceremony's two routes share one budget, so 30 a minute per address is 15 ceremonies, far above one person's retries and enough that a burst costs Postgres little. Like the one-time limits, they are keyed per address (an IPv6 /64), so they bound one caller, not a botnet. +## Relay sockets + +A `RelayRoom` that opened a Postgres connection through Hyperdrive could not be evicted afterwards: `unsafeEvictDurableObject` timed out on "it still has active references" even after the client had ended and its socket was closed, while an object that opened and closed a bare socket to the same host evicted normally (measured in Miniflare 5.20260908, 2026-10). Reading the rows in a Worker invocation of their own, through `ctx.exports`, leaves the object hibernatable. + ## Burrow enrollment - Begin stores nothing (2026-10-01): a stored request needed a cap, and `begin` is unauthenticated, so the cap could only be global, which a few /64s at the per-address limit could fill, locking every Burrow out of enrolling. With the expiry inside the device code and the user code derived from it, begin costs one HMAC and grows nothing. diff --git a/docs/specs/one-time.md b/docs/specs/one-time.md index cfc6e7751..965bc2d10 100644 --- a/docs/specs/one-time.md +++ b/docs/specs/one-time.md @@ -89,9 +89,10 @@ Each message is one JSON frame with exact keys, forwarded verbatim by the room: | `OneTimeBurrowFrame` | Burrow → phone | `{t: 'one-time', step: 'response' \| 'transport', ct}` | `ct` is one base64url Noise message, bounded as on the relay envelope. -`ONE_TIME_PING` / `ONE_TIME_PONG` are whole-string keepalives, never JSON: -each end pings its open socket every `ONE_TIME_PING_INTERVAL_MS` (30 s) and -counts no answer. +Each end keeps its open socket alive with the relay socket's heartbeat +(`docs/specs/relay.md` -> "Routing"): `RELAY_PING` every +`RELAY_PING_INTERVAL_MS` (30 s), whole strings never JSON, holding the room to +no deadline. - **Must measure a frame's raw text against `MAX_ONE_TIME_FRAME_LENGTH` before parsing it**; both ends read the room through `parseOneTimeFrame`. A room @@ -267,7 +268,7 @@ string verbatim (rationale). per 60 seconds. - **The room's state is the Burrow socket's hibernation attachment** — `expiresAt`, `joined`, and a count of every frame received — never memory, so - a hibernated room keeps its join and its count. `ONE_TIME_PING` is answered by + a hibernated room keeps its join and its count. `RELAY_PING` is answered by the runtime's auto-response and never wakes, forwards, or counts. A room's life, each step one event on the object: diff --git a/docs/specs/relay.md b/docs/specs/relay.md index 6a89a22d4..ed093b207 100644 --- a/docs/specs/relay.md +++ b/docs/specs/relay.md @@ -584,6 +584,7 @@ Its four resource bounds, enforced under `sweepRelaySockets` or at the socket: `MAX_E2E_CIPHERTEXT_LENGTH`, `MAX_CLIENT_ID_LENGTH` and `E2E_ID_LENGTH`, the same bounds the frame guards enforce — because `ws` otherwise buffers up to 100 MiB whole before `isE2eClientFrame` runs. Over it, the socket closes 1009. + **The bound is in UTF-8 bytes on every Relay**, the unit `maxPayload` counts. * **Client sockets are capped** at `MAX_RELAY_CLIENT_SOCKETS` (64), and the (n+1)-th **is refused, never admitted by evicting another**: a live socket belongs to a ceremony or an attached terminal, so evicting would let a token @@ -600,11 +601,20 @@ Its four resource bounds, enforced under `sweepRelaySockets` or at the socket: session just expired. * **A half-open connection is closed by heartbeat**, or its entry and its Burrow binding would live until the OS gave up: the sweep pings every socket and - closes whatever has not answered within `RELAY_IDLE_TIMEOUT_MS` (three sweeps) - with 1001; a message or pong refreshes liveness. **Must unregister both socket + closes whatever has not answered within `RELAY_IDLE_TIMEOUT_MS` (three ping + intervals) with 1001; a message or pong refreshes liveness. **Must unregister both socket kinds before starting the idle close handshake**, releasing routing and Client capacity immediately, pinned by `relay/test/relay-limits.test.mjs`. +**Must answer the text `RELAY_PING` with `RELAY_PONG`** on either socket kind, +compared whole before any parse, never forwarded and never an `error`. **The +Burrow and Pocket each send `RELAY_PING` every `RELAY_PING_INTERVAL_MS` (30 s) +and enforce a deadline only once a pong has arrived on that socket**: a ping +still unanswered when the next is due ends it, through that end's ordinary +close handling, so a Relay that never answers is never held to one. **Pocket +pauses its pings while the page is hidden** and forgives the outstanding one on +return, as its keepalives do. + `start.ts` runs the sweep every `RELAY_SWEEP_MS` (30 s), `unref`'d like the revocation sweep and far more often, touching no disk. Pinned by `relay/test/relay-limits.test.mjs`. @@ -619,9 +629,20 @@ the evicted Burrow keys its stand-down on the code (see socket's own close event is a no-op here, and the new Burrow process has a fresh ACL and no memory of them. -Source of truth: `relay/src/relay.ts` (`registerBurrow`), and `isE2eClientFrame` / -`isE2eBurrowFrame` in `remote-lib-common/src/remote/wire.ts`, written for a Burrow to -reuse verbatim. +**Every Relay reads and rebuilds a frame through one frame layer**: +`readClientFrame` / `readBurrowFrame`, which answer or drop as above, and +`toBurrowEnvelope` / `toClientEnvelope`, which rebuild it field by field. + +Source of truth: `relay/src/relay.ts` (`registerBurrow`, `answeredPing`); +`isE2eClientFrame` / `isE2eBurrowFrame` and `RELAY_PING` in +`remote-lib-common/src/remote/wire.ts`, written for a Burrow to reuse verbatim; +the frame layer in `remote-lib-common/src/remote/relay-routing.ts`; +`MAX_RELAY_FRAME_BYTES` / `MAX_RELAY_CLIENT_SOCKETS` / `RELAY_IDLE_TIMEOUT_MS` in +`remote-lib-common/src/remote/relay-common.ts`; `RelayHeartbeat` in +`lib/src/remote/ws.ts`. **Every Relay passes the same routing cases**, +`socketCases` / `e2eCases` in `remote-lib-common/test/harness/relay-parity.mjs`, +which `relay/test/relay.test.mjs` and `relay/test/e2e-relay.test.mjs` register +here and `hosted/server/tests/relay-room.test.ts` on Hosted. ### Pairing (phone ↔ laptop, first time) @@ -972,10 +993,10 @@ is the plaintext stub at `GET /`. ## Testing `pnpm --filter relay test` drives setup → pairing → connect through real HTTP -and WebSocket boundaries: the `FakeBurrow` in `relay/test/harness/fake-burrow.mjs` +and WebSocket boundaries: the `FakeBurrow` in `remote-lib-common/test/harness/fake-burrow.mjs` speaks only the `e2e` envelope and `client-gone`, mirroring the shipped Burrow's ceremony semantics over the same shared primitives; the `FakeClient` in -`relay/test/harness/fake-client.mjs` runs both ceremonies as a real Noise +`remote-lib-common/test/harness/fake-client.mjs` runs both ceremonies as a real Noise initiator, `SimAuthenticator` producing presence proofs through the real `/api/reauth/*` routes; process-level tests spawn the real entrypoint. `remote-lib-common/test/security-guarantees.test.mjs` drives the model's @@ -1130,17 +1151,11 @@ Unstaged but adjacent: origin migration (re-binding the passkey and enrollments after a Tailscale node rename), and the revocation UI staged in [remote-security-model.md](./remote-security-model.md) `## Future`. -**Scope: saas-multitenant** — the managed Relay on `relay.dormouse.sh` beyond the account-scoped routes, Pocket, and the device-code enrollment [hosted.md](./hosted.md) → "Relay" and "Burrow enrollment" serve, in order: tenant-scoped relay sockets, and push. The **remote-network** scope in [remote-network.md](./remote-network.md) owns the deployment, transport, and network restriction design. +**Scope: saas-multitenant** — the managed Relay on `relay.dormouse.sh` beyond the account-scoped routes, Pocket, the device-code enrollment, and the per-account relay sockets [hosted.md](./hosted.md) → "Relay", "Relay sockets", and "Burrow enrollment" serve: push. The **remote-network** scope in [remote-network.md](./remote-network.md) owns the deployment, transport, and network restriction design. ### From single-owner to multi-tenant Selfhost (everything above the fold) stays as-is; SaaS is a parallel deployment that lifts each single-tenant simplification, every one chosen to be liftable: -* **Relay tenant-scoping (an invariant, not a check).** The relay binds one Burrow - per Client socket with no notion of tenant; multi-tenant makes tenancy - intrinsic to that binding — a Client may only ever be offered, and bound to, - Burrows of its own account, and a cross-tenant binding must be *impossible*, not - merely unauthorized. Defense-in-depth: the Burrow still authorizes, but the relay - must not be the weak point. * **Hosted transport.** Follow the **remote-network** scope in [remote-network.md](./remote-network.md) for routing and lifecycle. diff --git a/docs/specs/remote-network.md b/docs/specs/remote-network.md index 940132ec9..6d5082b99 100644 --- a/docs/specs/remote-network.md +++ b/docs/specs/remote-network.md @@ -101,7 +101,7 @@ Source of truth: `NetworkSettings`, `NetworkPhones`, `NetworkUpdates`, `connecti **Scope: remote-network** — build in order: 1. **Anywhere on a phone**: **Must measure iOS Safari's offer size and gathering time, and the Burrow with STUN blocked**, before changing a budget. -2. **Hosted persistent**: desktop enrollment, the Hosted Relay's sockets, and push, beyond its routes in `docs/specs/hosted.md` -> "Relay" and "Burrow enrollment", with **saas-multitenant** in `docs/specs/relay.md`, its connections in `connectionsFor`; Local networks and Anywhere then cover paired phones. +2. **Hosted persistent**: desktop enrollment, the path rule for paired phones, and push, beyond the routes and sockets in `docs/specs/hosted.md` -> "Relay", "Relay sockets", and "Burrow enrollment", with **saas-multitenant** in `docs/specs/relay.md`, its connections in `connectionsFor`; Local networks and Anywhere then cover paired phones. ### Allowed networks @@ -109,8 +109,6 @@ Source of truth: `NetworkSettings`, `NetworkPhones`, `NetworkUpdates`, `connecti ### Hosted persistent -- **Must route account-scoped Relay sockets through Durable Objects**, keyed per Burrow, preserving bounded opaque frames and tenant isolation; Hosted login never authorizes a terminal. -- **Must use WebSocket hibernation**, rebuilding routing from attachments and durable metadata, and never store terminal ciphertext. - **Under Local networks a paired phone's session is direct-only**, with the one-time rule: an application message off the Relay ends it unread. **May fall back to Hosted relaying under Anywhere.** - **Must choose Pocket's direct-peer factory by deployment** ("Anywhere"), one bundle serving both; `lib/src/remote/pocket-app/App.tsx` hard-codes `selfHostDirectPeer`. - **Must start `BurrowRuntime` on the level's `directPeeringFor`, restarting it on any change `samePaths` sees** ("Anywhere"). diff --git a/docs/specs/remote-security-model.md b/docs/specs/remote-security-model.md index d8cef6d4c..533e0ec41 100644 --- a/docs/specs/remote-security-model.md +++ b/docs/specs/remote-security-model.md @@ -677,6 +677,9 @@ until then known only to the Relay — and one counter, so a Relay that kept an envelope can re-deliver it ([Push sealing](#push-sealing)). +**Hosted's Relay observes what the self-host Relay does**, each account's in +its own `RelayRoom` ([hosted.md](./hosted.md) -> "Relay sockets"). + **Hosted's one-time rendezvous observes each room's timing, both ends' IP addresses, the room id, and the size and count of the handshake frames it forwards** — never plaintext, and nothing once the session is direct diff --git a/docs/specs/security-hosted.md b/docs/specs/security-hosted.md index 419598ce8..cf2fd99d0 100644 --- a/docs/specs/security-hosted.md +++ b/docs/specs/security-hosted.md @@ -43,7 +43,7 @@ Pinned by `hosted/server/tests/workers.test.ts` and `hosted/server/tests/policy. ## Relay boundary -**The Hosted Relay** on the relay Worker, and its account routes on the account Worker: `docs/specs/hosted.md` -> "Relay" and "Burrow enrollment" own them; these are the checks on them. Inspect `relayApiRoutes` in `hosted/server/relay-api.ts`, `hosted/server/relay-auth.ts`, `relayAccountRoutes` in `hosted/server/relay-account.ts`, `cookieAdmin` in `hosted/server/account-gate.ts`, and `hosted/server/dormouse-migrations/002_relay.sql`. +**The Hosted Relay** on the relay Worker, and its account routes on the account Worker: `docs/specs/hosted.md` -> "Relay", "Relay sockets", and "Burrow enrollment" own them; these are the checks on them. Inspect `relayApiRoutes` in `hosted/server/relay-api.ts`, `hosted/server/relay-auth.ts`, `relaySocketRoutes` in `hosted/server/relay-sockets.ts`, `RelayRoom` in `hosted/server/relay-room.ts`, the frame layer in `remote-lib-common/src/remote/relay-routing.ts`, `relayAccountRoutes` in `hosted/server/relay-account.ts`, `cookieAdmin` in `hosted/server/account-gate.ts`, and `hosted/server/dormouse-migrations/002_relay.sql`. - **FAIL IF** a session, Burrow, or setup token is stored other than as its SHA-256, an enrollment device code is stored at all, or an account's Relay rows outlive its user row. - **FAIL IF** a query reading a Burrow, passkey, presence nonce, or setup token is not scoped to the caller's account (the session's user, or the Burrow token's owner), a passkey registers to any account but the minting Burrow's owner, or a setup challenge redeems with another Burrow's token. @@ -55,9 +55,13 @@ Pinned by `hosted/server/tests/workers.test.ts` and `hosted/server/tests/policy. - **FAIL IF** an approval admits a request without a login, with an `Origin` other than the account's own exactly, from a login older than `LOGIN_FRESH_AGE_MS` or one whose creation time it cannot read, for an account but the entitled admin, or past the per-account attempt limit (`RELAY_APPROVE_LIMIT`, counted before the body is read). - **FAIL IF** a table a caller can grow has no cap keyed by whoever grows it (approvals: the per-account attempt limit), or a capped write deletes another key's rows; `signin/*` or `setup/begin`/`finish` reaches the database before its per-address limit; or a production `namespace_id` reaches `PREVIEW_RATELIMIT_OFFSET`. - **FAIL IF** the relay Worker reads a cookie, asks auth, or reads a user column but the entitlement's; inspect the relay bundle's imports. +- **FAIL IF** `RelayRoom` stores, logs, or decodes a frame or its `ct`, reads a frame other than through the shared frame layer (`readClientFrame`, `readBurrowFrame`), which parses only the routing envelope through the shared guards and copies `ct` field by field and reads it nowhere else, parses a frame before measuring its UTF-8 bytes against the shared `MAX_RELAY_FRAME_BYTES`, or stores anything but its account id; `scripts/e2e-lint.mjs` holds the name, parse, decode, log, storage, and attachment half textually in both modules. +- **FAIL IF** a `RelayRoom` is named from anything but the account an authenticated token resolved to, or serves a request or RPC naming an account other than the one it stored. +- **FAIL IF** the client socket upgrade admits an `Origin` other than exactly the relay's `APP_ORIGIN`, the Burrow upgrade admits any `Origin`, either reaches the object before its token resolves to a live session or Burrow of an entitled account, or the Worker hands the object any header, token, or parameter of the caller's. +- **FAIL IF** the object accepts a Burrow socket without rechecking its row (enrolled, the account's, owner entitled) under `blockConcurrencyWhile`, or awaits any row read past `RELAY_ROW_READ_TIMEOUT_MS`, so a stalled database resets every socket of the account; a removed Burrow's live socket outlives its removal's `closeBurrow`, or a removed or de-entitled Burrow's outlives the next sweep, at most `RELAY_ROOM_SWEEP_MS` while any Burrow socket is held; or a route reaches `closeBurrow`, `onlineBurrows`, `RelayRows`, or any other `RelayRoom` method but the two upgrades. - **FAIL IF** a Pocket path's response lacks Pocket's policy (`pocketContentSecurityPolicy` in `remote-lib-common/src/remote/relay-common.ts`, taken only from an `APP_ORIGIN` that is exactly an origin), any response but a Pocket path's allows the camera, `/connect/` included, or a response is classified on any path but the decoded one Hono routes on, so `/%63onnect/` would take Pocket's; inspect `relayRules` / `relayPathKind` and `secureHeaders` in `hosted/server/headers.ts`. -Pinned by `hosted/server/tests/relay.test.ts`, `hosted/server/tests/pocket.test.ts`, and `hosted/server/tests/boundary.test.ts`. +Pinned by `hosted/server/tests/relay.test.ts`, `hosted/server/tests/relay-room.test.ts`, `hosted/server/tests/workers.test.ts`, `hosted/server/tests/pocket.test.ts`, and `hosted/server/tests/boundary.test.ts`. ## Deployment boundary diff --git a/hosted/scripts/preview.mjs b/hosted/scripts/preview.mjs index 78063a0c6..0395b310e 100644 --- a/hosted/scripts/preview.mjs +++ b/hosted/scripts/preview.mjs @@ -56,11 +56,18 @@ export function previewConfig(base, env, worker, hyperdriveId) { id: hyperdriveId, })); } - // Migrations travel only with the Durable Objects they create: a preview - // Worker that implements none starts with none to delete. + // A binding to another Worker's class names that Worker's preview, never + // production's. Migrations travel only with the Durable Objects a Worker + // implements: one that implements none starts with none to delete. if (base.durable_objects) { - config.durable_objects = base.durable_objects; - if (base.migrations) config.migrations = base.migrations; + const bindings = base.durable_objects.bindings.map((binding) => + binding.script_name + ? { ...binding, script_name: previewName(env.PR_NUMBER, binding.script_name) } + : binding, + ); + config.durable_objects = { ...base.durable_objects, bindings }; + if (base.migrations && bindings.some((binding) => !binding.script_name)) + config.migrations = base.migrations; } if (base.ratelimits) config.ratelimits = base.ratelimits.map((limit) => ({ diff --git a/hosted/scripts/preview.test.mjs b/hosted/scripts/preview.test.mjs index 1684fecd4..ab449829f 100644 --- a/hosted/scripts/preview.test.mjs +++ b/hosted/scripts/preview.test.mjs @@ -85,10 +85,13 @@ test("each preview configuration isolates its origin and excludes production bin for (const worker of ["account", "relay"]) assert.equal(configs[worker].assets.run_worker_first, true, worker); // Production's account keeps the migrations that deleted its old room; its - // preview, implementing no Durable Object, carries none. + // preview, implementing no Durable Object, carries none, and its binding to + // the relay's `RelayRoom` names this PR's relay preview. assert.ok(bases.account.migrations?.length); assert.equal(configs.account.migrations, undefined); - assert.equal(configs.account.durable_objects, undefined); + assert.deepEqual(configs.account.durable_objects, { + bindings: [{ name: "RELAY_ROOM", class_name: "RelayRoom", script_name: "dormouse-relay-pr-42" }], + }); assert.deepEqual(configs.relay.durable_objects, bases.relay.durable_objects); assert.deepEqual(configs.relay.migrations, bases.relay.migrations); for (const worker of ["relay", "account"]) @@ -137,6 +140,27 @@ test("preview configuration keeps its own Durable Objects and rate-limit namespa assert.deepEqual(config.ratelimits, [ { name: "LIMIT", namespace_id: "1007", simple: { limit: 1, period: 60 } }, ]); + // Another Worker's class: its preview's, and it carries this Worker's + // migrations only beside a class of its own. + const foreign = { name: "OTHER", class_name: "Other", script_name: "dormouse-other" }; + const previewForeign = { ...foreign, script_name: "dormouse-other-pr-42" }; + const base = { name: "dormouse-hosted", compatibility_date: "2026-01-01", migrations }; + const onlyForeign = previewConfig( + { ...base, durable_objects: { bindings: [foreign] } }, + env, + "account", + ); + assert.deepEqual(onlyForeign.durable_objects, { bindings: [previewForeign] }); + assert.equal(onlyForeign.migrations, undefined); + const both = previewConfig( + { ...base, durable_objects: { bindings: [...durable_objects.bindings, foreign] } }, + env, + "account", + ); + assert.deepEqual(both.durable_objects, { + bindings: [...durable_objects.bindings, previewForeign], + }); + assert.deepEqual(both.migrations, migrations); for (const bad of ["0", "1000", "-3", "x", "1.5"]) assert.throws(() => previewRatelimitNamespace(bad)); }); diff --git a/hosted/scripts/production.test.mjs b/hosted/scripts/production.test.mjs index 9627a234c..f8480a00b 100644 --- a/hosted/scripts/production.test.mjs +++ b/hosted/scripts/production.test.mjs @@ -206,13 +206,21 @@ test("the history sweep's cron is the voice Worker's, the relay's sweeps its exp // An absent `triggers` would leave a deployed schedule in place. assert.deepEqual(configs.account.triggers, { crons: [] }); }); -test("the rendezvous Durable Object is the relay's, each rate limit its Worker's, and Durable Object migrations are append-only", () => { +test("the Durable Objects are the relay's, each rate limit its Worker's, and Durable Object migrations are append-only", () => { assert.deepEqual(configs.relay.durable_objects, { - bindings: [{ name: "ONE_TIME_ROOM", class_name: "OneTimeRoom" }], + bindings: [ + { name: "ONE_TIME_ROOM", class_name: "OneTimeRoom" }, + { name: "RELAY_ROOM", class_name: "RelayRoom" }, + ], }); assert.deepEqual(configs.relay.migrations, [ { tag: "v1", new_sqlite_classes: ["OneTimeRoom"] }, + { tag: "v2", new_sqlite_classes: ["RelayRoom"] }, ]); + // The account reaches the relay's `RelayRoom` by name, implementing nothing. + assert.deepEqual(configs.account.durable_objects, { + bindings: [{ name: "RELAY_ROOM", class_name: "RelayRoom", script_name: configs.relay.name }], + }); assert.deepEqual( configs.relay.ratelimits.map(({ name, namespace_id }) => [name, namespace_id]), [ @@ -235,8 +243,7 @@ test("the rendezvous Durable Object is the relay's, each rate limit its Worker's { tag: "v1", new_sqlite_classes: ["OneTimeRoom"] }, { tag: "v2", deleted_classes: ["OneTimeRoom"] }, ]); - for (const worker of ["account", "voice"]) - assert.equal(configs[worker].durable_objects, undefined, worker); + assert.equal(configs.voice.durable_objects, undefined); assert.equal(configs.voice.ratelimits, undefined); assert.equal(configs.voice.migrations, undefined); }); diff --git a/hosted/server/account-app.ts b/hosted/server/account-app.ts index d4e8c2c48..58cd044bc 100644 --- a/hosted/server/account-app.ts +++ b/hosted/server/account-app.ts @@ -3,6 +3,7 @@ import { queryDatabase } from "pgstencil/postgres"; import type { AccountEnv } from "./bindings"; import { accountRules } from "./headers"; import { relayAccountRoutes, type RelayAccountHost } from "./relay-account"; +import { relayRoom } from "./relay-room-contract"; import { voiceTokenRoutes } from "./voice"; import { workerApp } from "./worker-app"; @@ -47,6 +48,7 @@ export function accountApp( databaseUrl: c.env.HYPERDRIVE.connectionString, auth: (request) => fetchAuth(request, c.env, c.executionCtx), approveLimit: c.env.RELAY_APPROVE_LIMIT, + closeBurrow: (userId, burrowId) => relayRoom(c.env.RELAY_ROOM, userId).closeBurrow(burrowId), }); voiceTokenRoutes(app, host); relayAccountRoutes(app, host); diff --git a/hosted/server/bindings.ts b/hosted/server/bindings.ts index b84e5ca12..2f37ac1d6 100644 --- a/hosted/server/bindings.ts +++ b/hosted/server/bindings.ts @@ -1,6 +1,7 @@ import type { BetterAuthWorkerBindings } from "@pgstencil/auth/better-auth-workers"; import { exactOrigin } from "./headers"; import { providerBindings } from "./policy"; +import type { RelayRoomRpc } from "./relay-room-contract"; // Each Worker's bindings mapper (`docs/specs/security-hosted.md` -> "Origin // boundary"): the only bindings that reach its routes, whatever else the @@ -22,6 +23,8 @@ export interface AccountEnv extends BetterAuthWorkerBindings, WorkerEnv { ASSETS: Assets; /** Enrollment approvals, per account. */ RELAY_APPROVE_LIMIT: RateLimit; + /** The relay Worker's `RelayRoom`s, which a Burrow's removal closes the socket of. */ + RELAY_ROOM: DurableObjectNamespace; EMAIL_FROM: string; POSTMARK_SERVER_TOKEN: string; OAUTH_PROVIDERS?: string; @@ -32,6 +35,8 @@ export interface RelayEnv extends WorkerEnv { ASSETS: Assets; HYPERDRIVE: { connectionString: string }; ONE_TIME_ROOM: DurableObjectNamespace; + /** One `RelayRoom` per account, holding its relay sockets. */ + RELAY_ROOM: DurableObjectNamespace; ONE_TIME_MINT_LIMIT: RateLimit; ONE_TIME_JOIN_LIMIT: RateLimit; RELAY_SIGNIN_LIMIT: RateLimit; @@ -58,6 +63,7 @@ export const accountBindings = (env: AccountEnv): AccountEnv => ({ HYPERDRIVE: env.HYPERDRIVE, ASSETS: env.ASSETS, RELAY_APPROVE_LIMIT: env.RELAY_APPROVE_LIMIT, + RELAY_ROOM: env.RELAY_ROOM, APP_ORIGIN: env.APP_ORIGIN, AUTH_SECRET: env.AUTH_SECRET, EMAIL_FROM: env.EMAIL_FROM, @@ -71,6 +77,7 @@ export const accountPreviewBindings = (env: AccountEnv): AccountEnv => ({ HYPERDRIVE: env.HYPERDRIVE, ASSETS: env.ASSETS, RELAY_APPROVE_LIMIT: env.RELAY_APPROVE_LIMIT, + RELAY_ROOM: env.RELAY_ROOM, APP_ORIGIN: env.APP_ORIGIN, AUTH_SECRET: env.AUTH_SECRET, BUILD_SHA: env.BUILD_SHA, @@ -90,6 +97,7 @@ export const relayBindings = (env: RelayEnv): RelayEnv => ({ APP_ORIGIN: env.APP_ORIGIN, BUILD_SHA: env.BUILD_SHA, ONE_TIME_ROOM: env.ONE_TIME_ROOM, + RELAY_ROOM: env.RELAY_ROOM, ONE_TIME_MINT_LIMIT: env.ONE_TIME_MINT_LIMIT, ONE_TIME_JOIN_LIMIT: env.ONE_TIME_JOIN_LIMIT, RELAY_SIGNIN_LIMIT: env.RELAY_SIGNIN_LIMIT, diff --git a/hosted/server/dev.ts b/hosted/server/dev.ts index 193b4f673..1634f5503 100644 --- a/hosted/server/dev.ts +++ b/hosted/server/dev.ts @@ -65,6 +65,8 @@ const host: RelayAccountHost = { databaseUrl, auth: (request: Request) => auth.app.fetch(request), approveLimit: devLimit(10), + // No relay runs here, so no Burrow holds a socket to close. + closeBurrow: async () => {}, }; voiceTokenRoutes(app, () => host); relayAccountRoutes(app, () => host); diff --git a/hosted/server/one-time-room.ts b/hosted/server/one-time-room.ts index 70780f34c..cc6e9629b 100644 --- a/hosted/server/one-time-room.ts +++ b/hosted/server/one-time-room.ts @@ -3,8 +3,8 @@ import { MAX_ONE_TIME_FRAME_LENGTH, ONE_TIME_EXPIRY_GRACE_MS, ONE_TIME_LINK_TTL_MS, - ONE_TIME_PING, - ONE_TIME_PONG, + RELAY_PING, + RELAY_PONG, ONE_TIME_ROOM_PARAM, ONE_TIME_WS_ROUTES, WS_CLOSE_ONE_TIME_DEADLINE, @@ -22,12 +22,11 @@ import { type OneTimeRoomFrame, } from "remote-lib-common"; +import { OPEN, refuseSocket } from "./socket-room"; + /** A socket's role, as its hibernation tag. */ type Role = "burrow" | "client"; -/** `WebSocket.OPEN`. */ -const OPEN = 1; - /** The room's whole state, held as the Burrow socket's attachment. */ interface RoomState { expiresAt: number; @@ -53,7 +52,7 @@ export class OneTimeRoom { // Answered by the runtime, so a keepalive neither wakes the room nor // reaches `webSocketMessage` to be forwarded or counted. ctx.setWebSocketAutoResponse( - new WebSocketRequestResponsePair(ONE_TIME_PING, ONE_TIME_PONG), + new WebSocketRequestResponsePair(RELAY_PING, RELAY_PONG), ); } @@ -76,15 +75,10 @@ export class OneTimeRoom { } async #open(roomId: string): Promise { - const { 0: client, 1: server } = new WebSocketPair(); // A room id is minted fresh for each Burrow socket, so a room opens once. if (this.#socket("burrow") || this.#socket("client")) - return this.#refuse( - client, - server, - WS_CLOSE_ONE_TIME_UNAVAILABLE, - WS_CLOSE_ONE_TIME_UNAVAILABLE_REASON, - ); + return refuseSocket(WS_CLOSE_ONE_TIME_UNAVAILABLE, WS_CLOSE_ONE_TIME_UNAVAILABLE_REASON); + const { 0: client, 1: server } = new WebSocketPair(); const { linkTtlMs, expiryGraceMs } = this.limits(); const expiresAt = Date.now() + linkTtlMs; await this.#ctx.storage.setAlarm(expiresAt + expiryGraceMs); @@ -105,31 +99,15 @@ export class OneTimeRoom { } #join(): Response { - const { 0: client, 1: server } = new WebSocketPair(); // Check and set with no await between them: the one join is decided here. const burrow = this.#socket("burrow"); const state = burrow && this.#state(burrow); if (!burrow || !state) - return this.#refuse( - client, - server, - WS_CLOSE_ONE_TIME_UNAVAILABLE, - WS_CLOSE_ONE_TIME_UNAVAILABLE_REASON, - ); - if (state.joined) - return this.#refuse( - client, - server, - WS_CLOSE_ONE_TIME_TAKEN, - WS_CLOSE_ONE_TIME_TAKEN_REASON, - ); + return refuseSocket(WS_CLOSE_ONE_TIME_UNAVAILABLE, WS_CLOSE_ONE_TIME_UNAVAILABLE_REASON); + if (state.joined) return refuseSocket(WS_CLOSE_ONE_TIME_TAKEN, WS_CLOSE_ONE_TIME_TAKEN_REASON); if (Date.now() > state.expiresAt) - return this.#refuse( - client, - server, - WS_CLOSE_ONE_TIME_EXPIRED, - WS_CLOSE_ONE_TIME_EXPIRED_REASON, - ); + return refuseSocket(WS_CLOSE_ONE_TIME_EXPIRED, WS_CLOSE_ONE_TIME_EXPIRED_REASON); + const { 0: client, 1: server } = new WebSocketPair(); burrow.serializeAttachment({ ...state, joined: true } satisfies RoomState); this.#ctx.acceptWebSocket(server, ["client"]); return new Response(null, { status: 101, webSocket: client }); @@ -178,22 +156,6 @@ export class OneTimeRoom { : this.#end(WS_CLOSE_ONE_TIME_EXPIRED, WS_CLOSE_ONE_TIME_EXPIRED_REASON)); } - /** - * Accept-then-close, so the refused end reads a code rather than a failed - * upgrade. Accepted outside hibernation: the socket is never the room's, so - * none of its events reach the handlers above. - */ - #refuse( - client: WorkerWebSocket, - server: WorkerWebSocket, - code: number, - reason: string, - ): Response { - server.accept(); - server.close(code, reason); - return new Response(null, { status: 101, webSocket: client }); - } - /** Close every socket the room holds and delete what it stored. */ async #end(code: number, reason: string) { for (const role of ["burrow", "client"] as const) diff --git a/hosted/server/one-time.ts b/hosted/server/one-time.ts index a3b508261..36ade219c 100644 --- a/hosted/server/one-time.ts +++ b/hosted/server/one-time.ts @@ -8,6 +8,7 @@ import { toBase64Url, } from "remote-lib-common"; import type { RelayEnv } from "./bindings"; +import { forwardUpgrade, isUpgrade, upgradeRequired } from "./socket-room"; type OneTimeContext = Context<{ Bindings: RelayEnv }>; @@ -19,7 +20,7 @@ type OneTimeContext = Context<{ Bindings: RelayEnv }>; */ export function oneTimeRoutes(app: Hono<{ Bindings: RelayEnv }>) { app.get(ONE_TIME_WS_ROUTES.burrow, async (c) => { - if (!upgrade(c)) return upgradeRequired(c); + if (!isUpgrade(c)) return upgradeRequired(c, "message"); // Every browser sends Origin on a WebSocket handshake and the Burrow's // native socket sends none, so refusing any Origin keeps web pages from // minting rooms. @@ -32,7 +33,7 @@ export function oneTimeRoutes(app: Hono<{ Bindings: RelayEnv }>) { return forward(c, ONE_TIME_WS_ROUTES.burrow, room); }); app.get(ONE_TIME_WS_ROUTES.client, async (c) => { - if (!upgrade(c)) return upgradeRequired(c); + if (!isUpgrade(c)) return upgradeRequired(c, "message"); if (c.req.raw.headers.get("origin") !== c.env.APP_ORIGIN) return c.json({ message: "Forbidden." }, 403); const rooms = new URL(c.req.url).searchParams.getAll(ONE_TIME_ROOM_PARAM); @@ -68,14 +69,6 @@ export function assetOrNotFound(c: OneTimeContext, response: Response) { : c.notFound(); } -function upgrade(c: OneTimeContext) { - return c.req.raw.headers.get("upgrade")?.toLowerCase() === "websocket"; -} - -function upgradeRequired(c: OneTimeContext) { - return c.json({ message: "WebSocket upgrade required." }, 426); -} - function tooMany(c: OneTimeContext) { return c.json({ message: "Too many requests." }, 429); } @@ -142,13 +135,8 @@ function ipv6Groups(ip: string): string[] { ]; } -/** - * A fresh request carrying only the upgrade and the room, so no cookie, - * address, or Origin reaches the room. - */ +/** The room's object, handed a bare upgrade naming the room and nothing else of the caller's. */ function forward(c: OneTimeContext, route: string, room: string) { - const url = new URL(route, c.req.url); - url.searchParams.set(ONE_TIME_ROOM_PARAM, room); const stub = c.env.ONE_TIME_ROOM.get(c.env.ONE_TIME_ROOM.idFromName(room)); - return stub.fetch(new Request(url, { headers: { upgrade: "websocket" } })); + return forwardUpgrade(stub, new URL(route, c.req.url), { [ONE_TIME_ROOM_PARAM]: room }); } diff --git a/hosted/server/relay-account.ts b/hosted/server/relay-account.ts index ad919f015..a5ce9a61e 100644 --- a/hosted/server/relay-account.ts +++ b/hosted/server/relay-account.ts @@ -14,6 +14,8 @@ import { ENROLLMENT_TTL_MS, LOGIN_FRESH_AGE_MS, RECENT_LOGIN_WINDOW } from "./po export type RelayAccountHost = AccountHost & { /** Approval attempts, keyed by account. */ approveLimit: RateLimit; + /** Close the removed Burrow's live relay socket (`RelayRoom.closeBurrow`). */ + closeBurrow(userId: string, burrowId: string): Promise; }; /** The 403 an approval from a login older than the recent-login window gets. */ @@ -83,18 +85,28 @@ export function relayAccountRoutes(app: Hono, host: (c: Context) => RelayAc }); // The row goes, and its setup tokens and setup challenges with it, so its - // Burrow token finds nothing on any relay route. + // Burrow token finds nothing on any relay route; then its live socket, if + // it holds one, is closed as revoked and its Clients told `burrow-gone`. + // The removal is done once the row is: a close that fails is logged, and + // the object's hourly sweep closes the socket instead. app.delete("/api/relay/burrows/:burrowId", gate, async (c) => { const burrowId = c.req.param("burrowId"); + const { userId } = c.get("login"); const removed = isE2eId(burrowId) && ( await accountQuery( host(c), `DELETE FROM dormouse_relay_burrows WHERE "burrowId" = $1 AND "userId" = $2 RETURNING 1`, - [burrowId, c.get("login").userId], + [burrowId, userId], ) ).length > 0; - return removed ? c.body(null, 204) : c.json({ message: "Computer not found." }, 404); + if (!removed) return c.json({ message: "Computer not found." }, 404); + try { + await host(c).closeBurrow(userId, burrowId); + } catch (error) { + console.error("Closing a removed Burrow's relay socket failed; the sweep will", error); + } + return c.body(null, 204); }); } diff --git a/hosted/server/relay-api.ts b/hosted/server/relay-api.ts index d2136a29b..0b94e2bb5 100644 --- a/hosted/server/relay-api.ts +++ b/hosted/server/relay-api.ts @@ -58,6 +58,7 @@ import type { import type { RelayEnv } from "./bindings"; import { allowed } from "./one-time"; import { ENROLLMENT_TTL_MS } from "./policy-constants"; +import { relayRoom } from "./relay-room-contract"; import { OWNER_COLUMNS, database, @@ -407,14 +408,18 @@ export function relayApiRoutes(app: Hono<{ Bindings: RelayEnv }>) { // --- Discovery and the Burrow's setup tokens ------------------------------ app.get(API_ROUTES.burrows, requireSession, async (c) => { - const { rows } = await c.var.db.query<{ burrowId: string }>( - `SELECT "burrowId" FROM dormouse_relay_burrows - WHERE "userId" = $1 ORDER BY "enrolledAt", "burrowId"`, - [c.var.session.userId], - ); - // No relay socket reaches this Worker yet, so none is connected. + const { userId } = c.var.session; + // The rows and the live sockets at once: neither waits on the other. + const [{ rows }, online] = await Promise.all([ + c.var.db.query<{ burrowId: string }>( + `SELECT "burrowId" FROM dormouse_relay_burrows + WHERE "userId" = $1 ORDER BY "enrolledAt", "burrowId"`, + [userId], + ), + relayRoom(c.env.RELAY_ROOM, userId).onlineBurrows(), + ]); const res: BurrowsResponse = { - burrows: rows.map(({ burrowId }) => ({ burrowId, online: false })), + burrows: rows.map(({ burrowId }) => ({ burrowId, online: online.includes(burrowId) })), }; return c.json(res); }); diff --git a/hosted/server/relay-auth.ts b/hosted/server/relay-auth.ts index 3792fb555..d1fde1c5a 100644 --- a/hosted/server/relay-auth.ts +++ b/hosted/server/relay-auth.ts @@ -28,6 +28,8 @@ export interface Owner { /** A live sign-in session, by its token's hash. */ export interface RelaySession extends Owner { tokenHash: string; + /** When it expires, epoch milliseconds: a relay socket it opens closes then. */ + expiresAt: number; } /** An enrolled Burrow. */ @@ -57,13 +59,14 @@ export async function sessionByToken(db: Client, token: string): Promise( - `SELECT s."userId", ${OWNER_COLUMNS} + } = await db.query( + `SELECT s."userId", ${OWNER_COLUMNS}, + floor(extract(epoch from s."expiresAt") * 1000)::float8 AS "expiresAt" FROM dormouse_relay_sessions s JOIN "user" u ON u.id = s."userId" WHERE s."tokenHash" = $1 AND s."expiresAt" > now()`, [tokenHash], ); - return row ? { ...ownerOf(row), tokenHash } : null; + return row ? { ...ownerOf(row), tokenHash, expiresAt: row.expiresAt } : null; } /** The Burrow `token` names, its owner joined in the same query, or null. */ @@ -79,6 +82,20 @@ export async function burrowByToken(db: Client, token: string): Promise { + const { rows } = await db.query( + `SELECT b."burrowId", b."userId", ${OWNER_COLUMNS} + FROM dormouse_relay_burrows b JOIN "user" u ON u.id = b."userId" + WHERE b."burrowId" = ANY($1::text[])`, + [burrowIds], + ); + return rows.map((row) => ({ ...ownerOf(row), burrowId: row.burrowId })); +} + /** A route's bindings plus what its credential gate resolved. */ export type RelayHonoEnv = { Bindings: RelayEnv; diff --git a/hosted/server/relay-room-contract.ts b/hosted/server/relay-room-contract.ts new file mode 100644 index 000000000..d8a4bf6d3 --- /dev/null +++ b/hosted/server/relay-room-contract.ts @@ -0,0 +1,47 @@ +// The contract between Hosted's Workers and the relay's `RelayRoom` +// (`docs/specs/hosted.md` -> "Relay sockets"): how an object is named, what an +// upgrade hands it, and the RPC either Worker may call. Apart from the object, +// so neither Worker's routes import it nor it theirs. + +/** + * The verified fields the relay Worker hands a `RelayRoom` with an upgrade, as + * search parameters on a request it builds fresh: the account the token + * named, and the Burrow's id or the session's expiry. + */ +export const RELAY_ROOM_PARAMS = { + account: "account", + burrowId: "burrow", + expiresAt: "expires", +} as const; + +/** + * The longest a `RelayRoom` holding a Burrow socket goes between re-reading + * its Burrows' rows: the backstop that closes a removed or de-entitled Burrow + * when the removal's own close did not arrive. + */ +export const RELAY_ROOM_SWEEP_MS = 60 * 60 * 1000; + +/** + * The longest a `RelayRoom` waits on a read of its Burrows' rows before + * treating it as failed: well under the 30 s after which the runtime resets + * an object whose `blockConcurrencyWhile` callback has not settled. + */ +export const RELAY_ROW_READ_TIMEOUT_MS = 5_000; + +/** The RPC a `RelayRoom` serves. Each names the account, which the object checks against its own. */ +export interface RelayRoomRpc { + /** Close `burrowId`'s socket as revoked (4001), its Clients told `burrow-gone`; whether one was held. */ + closeBurrow(account: string, burrowId: string): Promise; + /** Every Burrow holding a live socket: what `GET /api/burrows` reports online. */ + onlineBurrows(account: string): Promise; +} + +/** The account's `RelayRoom`, named from its user id and nothing else, its RPC bound to that account. */ +export function relayRoom(namespace: DurableObjectNamespace, account: string) { + const stub = namespace.get(namespace.idFromName(account)); + return { + fetch: (request: Request) => stub.fetch(request), + closeBurrow: (burrowId: string) => stub.closeBurrow(account, burrowId), + onlineBurrows: () => stub.onlineBurrows(account), + }; +} diff --git a/hosted/server/relay-room.ts b/hosted/server/relay-room.ts new file mode 100644 index 000000000..ff86bff3b --- /dev/null +++ b/hosted/server/relay-room.ts @@ -0,0 +1,443 @@ +// Rules: docs/specs/hosted.md -> "Relay sockets"; the routing it shares with +// the self-host Relay, docs/specs/relay.md -> "Routing"; what it may read and +// keep, docs/specs/security-hosted.md -> "Relay boundary", which +// scripts/e2e-lint.mjs holds textually here. +import { DurableObject } from "cloudflare:workers"; +import { + MAX_RELAY_CLIENT_SOCKETS, + MAX_RELAY_FRAME_BYTES, + NOT_ENTITLED_ERROR, + RELAY_IDLE_TIMEOUT_MS, + RELAY_PING, + RELAY_PONG, + UNKNOWN_BURROW_TOKEN_ERROR, + WS_CLOSE_BURROW_REPLACED, + WS_CLOSE_BURROW_REPLACED_REASON, + WS_CLOSE_BURROW_REVOKED, + WS_CLOSE_BURROW_REVOKED_REASON, + WS_CLOSE_FRAME_TOO_LARGE, + WS_CLOSE_IDLE, + WS_CLOSE_IDLE_REASON, + WS_CLOSE_TRY_AGAIN_LATER, + WS_CLOSE_UNAUTHORIZED, + WS_CLOSE_UNAUTHORIZED_REASON, + WS_ROUTES, + exceedsRelayFrameBytes, + newClientId, + offlineError, + readBurrowFrame, + readClientFrame, + toBurrowEnvelope, + toClientEnvelope, + type RelayToBurrowFrame, + type RelayToClientFrame, +} from "remote-lib-common"; +import type { RelayBurrow } from "./relay-auth"; +import { + RELAY_ROOM_PARAMS, + RELAY_ROOM_SWEEP_MS, + RELAY_ROW_READ_TIMEOUT_MS, + type RelayRoomRpc, +} from "./relay-room-contract"; +import type { RelayRows } from "./relay-rows"; +import { OPEN, refuseSocket } from "./socket-room"; + +/** The key of the object's one durable value: the account it serves, written once. */ +const ACCOUNT_KEY = "account"; + +/** + * Everything routing needs of a socket, held as its attachment so a woken + * object rebuilds it from `ctx.getWebSockets` rather than from memory. + * `retired` is the generation guard: set on a socket unregistered ahead of its + * close handshake, after which it is neither routed nor torn down again. + */ +type Conn = + | { role: "burrow"; burrowId: string; retired?: true } + | { + role: "client"; + clientId: string; + expiresAt: number; + burrowId: string | null; + retired?: true; + }; +type BurrowConn = Extract; +type ClientConn = Extract; + +/** A socket and its attachment. */ +type Held = { ws: WorkerWebSocket; conn: C }; + +/** Every socket carries its role as one tag, and its id as the other. */ +const burrowTag = (burrowId: string) => `burrow:${burrowId}`; +const clientTag = (clientId: string) => `client:${clientId}`; + +/** + * One account's Hosted Relay: that account's Burrow and Client sockets, + * routed by `docs/specs/relay.md` -> "Routing" exactly as the self-host + * `RelayHub` routes them, through the same frame layer. The relay Worker names + * it from the account an authenticated token resolved to, and it serves that + * account alone. Every socket is accepted through the Hibernation API, and a + * ping is answered by the runtime without waking it. + */ +export class RelayRoom extends DurableObject implements RelayRoomRpc { + constructor(ctx: DurableObjectState, env: unknown) { + super(ctx, env); + ctx.setWebSocketAutoResponse(new WebSocketRequestResponsePair(RELAY_PING, RELAY_PONG)); + } + + /** The clock expiry and silence are judged on; a test entry moves it. */ + protected now(): number { + return Date.now(); + } + + // Reached only through the relay Worker's socket routes, which have + // resolved the token, its account's entitlement, and the Origin. + async fetch(request: Request): Promise { + const url = new URL(request.url); + const param = (name: string) => url.searchParams.get(name) ?? ""; + const account = param(RELAY_ROOM_PARAMS.account); + const role = + url.pathname === WS_ROUTES.burrow ? "burrow" : url.pathname === WS_ROUTES.client ? "client" : null; + if (!role) return new Response(null, { status: 404 }); + if (!(await this.#claim(account))) return new Response(null, { status: 403 }); + return role === "burrow" + ? this.#openBurrow(account, param(RELAY_ROOM_PARAMS.burrowId)) + : this.#openClient(Number(param(RELAY_ROOM_PARAMS.expiresAt))); + } + + // --- RPC, from the Workers ------------------------------------------------ + + async closeBurrow(account: string, burrowId: string): Promise { + if (!(await this.#serves(account))) return false; + const held = this.#burrow(burrowId); + if (!held) return false; + this.#retire(held, WS_CLOSE_BURROW_REVOKED, WS_CLOSE_BURROW_REVOKED_REASON); + return true; + } + + async onlineBurrows(account: string): Promise { + if (!(await this.#serves(account))) return []; + return this.#held("burrow") + .filter((held) => !this.#goneSilent(held)) + .map(({ conn }) => conn.burrowId); + } + + // --- Opening ---------------------------------------------------------------- + + /** + * A Burrow socket, once its row says it may hold one: still enrolled, this + * account's, its owner entitled — refused as the Worker refuses a token + * otherwise. The Worker's check ran before this request was queued, so a + * removal committed since is caught here. Run under `blockConcurrencyWhile` + * so that removal's `closeBurrow`, sent after its commit, is delivered only + * once the socket is accepted and never between this read and the accept. + * + * One socket per `burrowId`: a second displaces the first — its Clients + * told `burrow-gone` and their bindings cleared now, since the displaced + * socket's own close is a no-op — and the first closes 4000. + */ + #openBurrow(account: string, burrowId: string): Promise { + return this.ctx.blockConcurrencyWhile(async () => { + let row: RelayBurrow | undefined; + try { + [row] = await this.#rows([burrowId]); + } catch { + console.error("RelayRoom could not read a Burrow's row"); + return new Response(null, { status: 503 }); + } + if (row?.userId !== account) + return Response.json({ error: UNKNOWN_BURROW_TOKEN_ERROR }, { status: 401 }); + if (!row.entitled) return Response.json({ error: NOT_ENTITLED_ERROR }, { status: 403 }); + const existing = this.#burrow(burrowId); + if (existing) + this.#retire(existing, WS_CLOSE_BURROW_REPLACED, WS_CLOSE_BURROW_REPLACED_REASON); + const { 0: client, 1: server } = new WebSocketPair(); + this.ctx.acceptWebSocket(server, ["burrow", burrowTag(burrowId)]); + server.serializeAttachment({ role: "burrow", burrowId } satisfies BurrowConn); + await this.#armBy(this.now() + RELAY_ROOM_SWEEP_MS); + return new Response(null, { status: 101, webSocket: client }); + }); + } + + /** + * A Client socket with a fresh secret `clientId`, or 1013 at the cap — + * refused, never admitted by evicting a live one. A Client silent past the + * idle timeout is not live, so one is retired here first; only at the cap, + * since judging silence costs a close. + */ + async #openClient(expiresAt: number): Promise { + if (!(expiresAt > this.now())) + return refuseSocket(WS_CLOSE_UNAUTHORIZED, WS_CLOSE_UNAUTHORIZED_REASON); + let clients = this.#held("client"); + if (clients.length >= MAX_RELAY_CLIENT_SOCKETS) + clients = clients.filter((held) => !this.#goneSilent(held)); + if (clients.length >= MAX_RELAY_CLIENT_SOCKETS) + return refuseSocket(WS_CLOSE_TRY_AGAIN_LATER, "too many client sockets"); + const clientId = newClientId(); + const { 0: client, 1: server } = new WebSocketPair(); + this.ctx.acceptWebSocket(server, ["client", clientTag(clientId)]); + server.serializeAttachment({ + role: "client", + clientId, + expiresAt, + burrowId: null, + } satisfies ClientConn); + await this.#armBy(expiresAt); + return new Response(null, { status: 101, webSocket: client }); + } + + // --- Frames ----------------------------------------------------------------- + + async webSocketMessage(ws: WorkerWebSocket, message: string | ArrayBuffer) { + const held = this.#registered(ws); + if (!held) return; + // Bounded before anything reads it, in the UTF-8 bytes the self-host + // Relay's `maxPayload` counts: the size of the text, never its content. + const oversized = + typeof message === "string" + ? exceedsRelayFrameBytes(message) + : message.byteLength > MAX_RELAY_FRAME_BYTES; + if (oversized) return this.#retire(held, WS_CLOSE_FRAME_TOO_LARGE, "frame too large"); + if (typeof message !== "string") return; + if (held.conn.role === "client") this.#onClientFrame(held as Held, message); + else this.#onBurrowFrame(held as Held, message); + } + + #onClientFrame({ ws, conn }: Held, raw: string) { + const read = readClientFrame(raw); + if ("error" in read) return send(ws, read.error); + const { frame } = read; + // A Burrow silent past the idle timeout is offline; judged here, where a + // frame would otherwise be routed into a dead socket. + const burrow = this.#burrow(frame.burrowId); + if (!burrow || this.#goneSilent(burrow)) return send(ws, offlineError(frame.burrowId)); + if (frame.step === "init") { + // An `init` binds, telling the Burrow it replaces that this Client is gone. + if (conn.burrowId !== null && conn.burrowId !== frame.burrowId) { + const previous = this.#burrow(conn.burrowId); + if (previous) send(previous.ws, clientGone(conn.clientId)); + } + conn.burrowId = frame.burrowId; + ws.serializeAttachment(conn); + } else if (conn.burrowId !== frame.burrowId) return; + send(burrow.ws, toBurrowEnvelope(conn.clientId, frame)); + } + + #onBurrowFrame({ conn }: Held, raw: string) { + const frame = readBurrowFrame(raw); + if (!frame) return; + const client = this.#client(frame.clientId); + // Only to a Client bound to this Burrow: a late reply from a Burrow the + // Client has left goes nowhere. + if (!client || client.conn.burrowId !== conn.burrowId) return; + send(client.ws, toClientEnvelope(conn.burrowId, frame)); + } + + // --- Closing ---------------------------------------------------------------- + + // A socket the object retired is already torn down; only a registered one + // tears down here. The alarm is left as it is: `alarm()` re-arms or clears. + async webSocketClose(ws: WorkerWebSocket, code: number) { + // Closing by now, so judged by its attachment alone. + const held = this.#attached(ws); + if (held) this.#unregister(held); + try { + ws.close(code === 1005 || code === 1006 ? 1000 : code); + } catch { + // The runtime already answered the close. + } + } + + async webSocketError(ws: WorkerWebSocket) { + await this.webSocketClose(ws, 1006); + } + + /** + * Every Client whose session has expired closes 1008, its Burrow told + * `client-gone`; while Burrow sockets are held, each is swept against its + * row. Then the alarm moves to the earlier of the next expiry and the next + * sweep, or is cleared while nothing is held. Opening a socket only ever + * brings the alarm earlier and a close leaves it, so an alarm may find + * nothing to do. + */ + async alarm() { + const now = this.now(); + for (const held of this.#held("client")) + if (held.conn.expiresAt <= now) + this.#retire(held, WS_CLOSE_UNAUTHORIZED, WS_CLOSE_UNAUTHORIZED_REASON); + if (this.#held("burrow").length > 0) await this.#sweep(); + const next = this.#held("client").map(({ conn }) => conn.expiresAt); + if (this.#held("burrow").length > 0) next.push(this.now() + RELAY_ROOM_SWEEP_MS); + if (next.length > 0) await this.ctx.storage.setAlarm(Math.min(...next)); + else await this.ctx.storage.deleteAlarm(); + } + + /** + * The backstop to `closeBurrow`: every held Burrow whose row is gone, is + * another account's, or whose owner is no longer entitled closes 4001, its + * Clients told `burrow-gone`. One query for them all; a failed or timed-out + * read leaves them to the next sweep. + */ + async #sweep() { + const account = await this.ctx.storage.get(ACCOUNT_KEY); + const burrowIds = this.#held("burrow").map(({ conn }) => conn.burrowId); + let rows: RelayBurrow[]; + try { + rows = await this.#rows(burrowIds); + } catch { + console.error("RelayRoom could not sweep its Burrows"); + return; + } + const standing = new Set( + rows.filter((row) => row.userId === account && row.entitled).map((row) => row.burrowId), + ); + for (const burrowId of burrowIds) { + if (standing.has(burrowId)) continue; + const held = this.#burrow(burrowId); + if (held) this.#retire(held, WS_CLOSE_BURROW_REVOKED, WS_CLOSE_BURROW_REVOKED_REASON); + } + } + + /** The alarm at `time`, unless one is already set sooner. */ + async #armBy(time: number) { + const current = await this.ctx.storage.getAlarm(); + if (current === null || time < current) await this.ctx.storage.setAlarm(time); + } + + /** + * Those of `burrowIds` still enrolled, with their owners, read by the + * Worker's own `RelayRows` in an invocation of its own: an object that has + * held a database connection is never evicted (rationale). Rejects past + * `RELAY_ROW_READ_TIMEOUT_MS`, so a stalled read under + * `blockConcurrencyWhile` fails as a failed read does, never resetting the + * object and every socket it holds. + */ + async #rows(burrowIds: string[]): Promise { + const { RelayRows } = this.ctx.exports as { RelayRows: Pick }; + let timer: ReturnType | undefined; + const timeout = new Promise((_, reject) => { + timer = setTimeout( + () => reject(new Error("RelayRows read timed out")), + RELAY_ROW_READ_TIMEOUT_MS, + ); + }); + try { + return await Promise.race([RelayRows.burrows(burrowIds), timeout]); + } finally { + clearTimeout(timer); + } + } + + /** + * Unregister, then start the close handshake: routing and capacity are + * released before the handshake, so a frame already buffered acts through + * nothing. + */ + #retire(held: Held, code: number, reason: string) { + this.#unregister(held); + try { + held.ws.close(code, reason); + } catch { + // Already closing. + } + } + + /** + * The teardown a disconnect performs: a Burrow's Clients get `burrow-gone` + * and lose their binding; a Client's Burrow gets `client-gone`. The socket's + * attachment is marked retired first, so a second teardown is a no-op. + */ + #unregister({ ws, conn }: Held) { + conn.retired = true; + try { + ws.serializeAttachment(conn); + } catch { + // A socket the runtime already closed keeps no attachment to protect. + } + if (conn.role === "burrow") { + for (const client of this.#held("client")) { + if (client.conn.burrowId !== conn.burrowId) continue; + send(client.ws, { t: "burrow-gone" } satisfies RelayToClientFrame); + client.conn.burrowId = null; + client.ws.serializeAttachment(client.conn); + } + } else if (conn.burrowId !== null) { + const burrow = this.#burrow(conn.burrowId); + if (burrow) send(burrow.ws, clientGone(conn.clientId)); + } + } + + /** + * Whether `held` stopped answering pings: its last auto-response is older + * than `RELAY_IDLE_TIMEOUT_MS`, and it is retired with 1001 here. A socket + * that never pinged is not judged. Checked where a dead socket would cost + * something — routing to a Burrow, admitting at the Client cap, reporting a + * Burrow online — and nowhere on a timer. + */ + #goneSilent(held: Held): boolean { + const answered = this.ctx.getWebSocketAutoResponseTimestamp(held.ws); + if (answered === null || this.now() - answered.getTime() <= RELAY_IDLE_TIMEOUT_MS) + return false; + this.#retire(held, WS_CLOSE_IDLE, WS_CLOSE_IDLE_REASON); + return true; + } + + // --- What the object holds ---------------------------------------------------- + + /** `ws` and its attachment while it is registered: open and not retired. */ + #registered(ws: WorkerWebSocket): Held | null { + return ws.readyState === OPEN ? this.#attached(ws) : null; + } + + /** `ws` and its attachment unless the object retired it. */ + #attached(ws: WorkerWebSocket): Held | null { + const conn = ws.deserializeAttachment() as Conn | null; + return conn && !conn.retired ? { ws, conn } : null; + } + + #held(role: R, tag: string = role) { + return this.ctx + .getWebSockets(tag) + .map((ws) => this.#registered(ws)) + .filter((held): held is Held> => held?.conn.role === role); + } + + #burrow(burrowId: string): Held | undefined { + return this.#held("burrow", burrowTag(burrowId))[0]; + } + + #client(clientId: string): Held | undefined { + return this.#held("client", clientTag(clientId))[0]; + } + + /** + * Admit `account` to this object: the first upgrade writes it, and one + * naming any other is refused. The Worker names the object from that same + * account, so a refusal here is an invariant broken upstream. + */ + async #claim(account: string): Promise { + if (!account) return false; + if ((await this.ctx.storage.get(ACCOUNT_KEY)) === undefined) + await this.ctx.storage.put(ACCOUNT_KEY, account); + return this.#serves(account); + } + + /** Whether this object serves `account`; one that never held a socket serves none. */ + async #serves(account: string): Promise { + const stored = await this.ctx.storage.get(ACCOUNT_KEY); + if (stored === undefined) return false; + if (stored === account) return true; + console.error("RelayRoom refused a request for another account"); + return false; + } +} + +const clientGone = (clientId: string): RelayToBurrowFrame => ({ t: "client-gone", clientId }); + +/** Serialize and send, swallowing errors from a socket that is mid-close. */ +function send(ws: WorkerWebSocket, frame: RelayToClientFrame | RelayToBurrowFrame) { + try { + ws.send(JSON.stringify(frame)); + } catch { + // The peer vanished between the lookup and this send. + } +} diff --git a/hosted/server/relay-rows.ts b/hosted/server/relay-rows.ts new file mode 100644 index 000000000..28948c2c7 --- /dev/null +++ b/hosted/server/relay-rows.ts @@ -0,0 +1,19 @@ +// Rules: docs/specs/hosted.md -> "Relay sockets". +import { WorkerEntrypoint } from "cloudflare:workers"; +import { withClient } from "pgstencil/postgres"; +import type { RelayEnv } from "./bindings"; +import { burrowsById, type Client, type RelayBurrow } from "./relay-auth"; + +/** + * The relay Worker's database read for its own `RelayRoom`s, reached through + * `ctx.exports` as a Worker invocation of its own, so an object never holds a + * database connection. A named entrypoint: no route and no binding reaches it. + */ +export class RelayRows extends WorkerEntrypoint> { + /** Those of `burrowIds` still enrolled, each with its owner. */ + burrows(burrowIds: string[]): Promise { + return withClient(this.env.HYPERDRIVE.connectionString, (db: Client) => + burrowsById(db, burrowIds), + ); + } +} diff --git a/hosted/server/relay-sockets.ts b/hosted/server/relay-sockets.ts new file mode 100644 index 000000000..f32494cf2 --- /dev/null +++ b/hosted/server/relay-sockets.ts @@ -0,0 +1,74 @@ +// Rules: docs/specs/hosted.md -> "Relay sockets"; docs/specs/security-hosted.md -> "Relay boundary". +import type { Context, Hono } from "hono"; +import { withClient } from "pgstencil/postgres"; +import { + NOT_ENTITLED_ERROR, + UNAUTHORIZED_ERROR, + UNKNOWN_BURROW_TOKEN_ERROR, + WS_ROUTES, + WS_TOKEN_PARAM, + isRelayBearer, +} from "remote-lib-common"; +import type { RelayEnv } from "./bindings"; +import { burrowByToken, sessionByToken, type Client } from "./relay-auth"; +import { RELAY_ROOM_PARAMS, relayRoom } from "./relay-room-contract"; +import { forwardUpgrade, isUpgrade, upgradeRequired } from "./socket-room"; + +type SocketContext = Context<{ Bindings: RelayEnv }>; + +/** + * The relay sockets, at the self-host Relay's paths and refusals: each + * resolves its token, then hands the account's `RelayRoom` a bare upgrade + * carrying only what the token resolved to. Register before the `/ws/*` tail. + */ +export function relaySocketRoutes(app: Hono<{ Bindings: RelayEnv }>) { + app.get(WS_ROUTES.burrow, async (c) => { + // Only a Node Burrow connects: every browser sends Origin on a WebSocket + // handshake, so refusing any keeps web pages off the Burrow socket. + if (c.req.raw.headers.has("origin")) return c.json({ error: "forbidden" }, 403); + if (!isUpgrade(c)) return upgradeRequired(c, "error"); + const burrow = await lookup(c, burrowByToken); + if (!burrow) return c.json({ error: UNKNOWN_BURROW_TOKEN_ERROR }, 401); + if (!burrow.entitled) return c.json({ error: NOT_ENTITLED_ERROR }, 403); + return forward(c, WS_ROUTES.burrow, burrow.userId, { + [RELAY_ROOM_PARAMS.burrowId]: burrow.burrowId, + }); + }); + + app.get(WS_ROUTES.client, async (c) => { + // Pocket is same-origin; any other page is a cross-site socket hijack. + if (c.req.raw.headers.get("origin") !== c.env.APP_ORIGIN) + return c.json({ error: "forbidden" }, 403); + if (!isUpgrade(c)) return upgradeRequired(c, "error"); + // An expired, unknown, or de-entitled session is one 401, which Pocket's + // probe of `GET /api/burrows` then reads as expiry. + const session = await lookup(c, sessionByToken); + if (!session?.entitled) return c.json({ error: UNAUTHORIZED_ERROR }, 401); + return forward(c, WS_ROUTES.client, session.userId, { + [RELAY_ROOM_PARAMS.expiresAt]: String(session.expiresAt), + }); + }); +} + +/** What the token query parameter names, refused by shape before any database read. */ +async function lookup( + c: SocketContext, + find: (db: Client, token: string) => Promise, +): Promise { + const token = c.req.query(WS_TOKEN_PARAM); + if (!isRelayBearer(token)) return null; + return withClient(c.env.HYPERDRIVE.connectionString, (db: Client) => find(db, token)); +} + +/** The account's object, handed a bare upgrade carrying the account and `fields`. */ +function forward( + c: SocketContext, + route: string, + account: string, + fields: Record, +) { + return forwardUpgrade(relayRoom(c.env.RELAY_ROOM, account), new URL(route, c.env.APP_ORIGIN), { + [RELAY_ROOM_PARAMS.account]: account, + ...fields, + }); +} diff --git a/hosted/server/relay-worker.ts b/hosted/server/relay-worker.ts index afea19c13..444f051e0 100644 --- a/hosted/server/relay-worker.ts +++ b/hosted/server/relay-worker.ts @@ -4,6 +4,7 @@ import { RELAY_NON_PAGE_PREFIXES, relayRules } from "./headers"; import { oneTimePageRoutes, oneTimeRoutes } from "./one-time"; import { pocketRoutes } from "./pocket"; import { relayApiRoutes, sweepExpired } from "./relay-api"; +import { relaySocketRoutes } from "./relay-sockets"; import { workerApp } from "./worker-app"; /** @@ -21,6 +22,7 @@ export default workerApp({ unavailable: "The relay is temporarily unavailable. Please try again.", routes(app) { relayApiRoutes(app); + relaySocketRoutes(app); oneTimeRoutes(app); oneTimePageRoutes(app); }, @@ -32,3 +34,5 @@ export default workerApp({ }); export { OneTimeRoom } from "./one-time-room"; +export { RelayRoom } from "./relay-room"; +export { RelayRows } from "./relay-rows"; diff --git a/hosted/server/socket-room.ts b/hosted/server/socket-room.ts new file mode 100644 index 000000000..71ca7a996 --- /dev/null +++ b/hosted/server/socket-room.ts @@ -0,0 +1,45 @@ +// What the relay Worker's two socket families share: the one-time rendezvous +// (`one-time.ts`, `one-time-room.ts`) and the relay sockets (`relay-sockets.ts`, +// `relay-room.ts`). Each route checks its request, then hands its Durable +// Object a fresh upgrade; each object refuses by closing an accepted socket. +import type { Context } from "hono"; + +/** `WebSocket.OPEN`. */ +export const OPEN = 1; + +/** Whether the request asks for a WebSocket upgrade. */ +export function isUpgrade(c: Context): boolean { + return c.req.raw.headers.get("upgrade")?.toLowerCase() === "websocket"; +} + +/** The 426 a socket route answers a plain request with, under its family's error key. */ +export function upgradeRequired(c: Context, key: "error" | "message"): Response { + return c.json({ [key]: "WebSocket upgrade required." }, 426); +} + +/** + * A fresh request carrying only the upgrade and `params`, so no header, + * token, cookie, or address of the caller's reaches the object. + */ +export function forwardUpgrade( + stub: { fetch(request: Request): Promise }, + url: URL, + params: Record, +): Promise { + const target = new URL(url); + target.search = ""; + for (const [name, value] of Object.entries(params)) target.searchParams.set(name, value); + return stub.fetch(new Request(target, { headers: { upgrade: "websocket" } })); +} + +/** + * Accept-then-close, so the refused end reads a code rather than a failed + * upgrade. Accepted outside hibernation: the socket is never the object's, so + * none of its events reach the object's handlers. + */ +export function refuseSocket(code: number, reason: string): Response { + const { 0: client, 1: server } = new WebSocketPair(); + server.accept(); + server.close(code, reason); + return new Response(null, { status: 101, webSocket: client }); +} diff --git a/hosted/server/tests/boundary.test.ts b/hosted/server/tests/boundary.test.ts index 1c64aab69..a2b3f2aee 100644 --- a/hosted/server/tests/boundary.test.ts +++ b/hosted/server/tests/boundary.test.ts @@ -7,6 +7,7 @@ import { API_ROUTES, ONE_TIME_PAGE_PATH, ONE_TIME_WS_ROUTES, + WS_ROUTES, } from "remote-lib-common"; import { accountBindings, @@ -25,12 +26,14 @@ import { import { ADMIN_EMAIL } from "../admin"; import { cookieAdmin } from "../account-gate"; import { RECENT_LOGIN_REQUIRED, relayAccountRoutes } from "../relay-account"; +import type { RelayRoomRpc } from "../relay-room-contract"; import { voiceApp } from "../voice-app"; import { workerApp } from "../worker-app"; import { ENTRIES, NAMES, ORIGINS, + alone, bundleWorker, miniflareOptions, type Name, @@ -69,7 +72,7 @@ beforeAll(async () => { NAMES.map(async (name) => { bundles[name] = await bundleWorker(ENTRIES[name]); workers[name] = new Miniflare( - miniflareOptions(name, bundles[name].outputFiles[0].text, { + alone(name, miniflareOptions(name, bundles[name].outputFiles[0].text, { bindings: { ...everything, APP_ORIGIN: ORIGINS[name] }, // Nothing listens here, so any database access would fail the request. hyperdrives: { HYPERDRIVE: "postgres://user:pass@127.0.0.1:9/none" }, @@ -92,7 +95,7 @@ beforeAll(async () => { return WorkerResponse.json({ history: [] }); return new WorkerResponse("{}", { status: 500 }); }, - }), + })), ); await workers[name].ready; }), @@ -126,6 +129,8 @@ const SERVED: Record = { ["GET", ONE_TIME_WS_ROUTES.burrow], ["GET", ONE_TIME_WS_ROUTES.client], ["GET", ONE_TIME_PAGE_PATH], + ["GET", WS_ROUTES.burrow], + ["GET", WS_ROUTES.client], ["POST", API_ROUTES.setupBegin], ["POST", API_ROUTES.signinBegin], ["GET", API_ROUTES.burrows], @@ -194,8 +199,7 @@ const ABSENT: Record = { ["GET", "/api/push/devices"], // The self-host installers' probe; neither a Burrow nor Pocket asks Hosted for it. ["GET", "/api/hello"], - // No relay socket is served yet, and the non-page prefixes 404 whole. - ["GET", "/ws/client"], + // The non-page prefixes 404 whole. ["GET", "/ws/other"], ["GET", "/api"], ["GET", "/ws"], @@ -214,6 +218,8 @@ const ABSENT: Record = { ["GET", "/login"], ...RELAY_API, ...ACCOUNT_RELAY, + ["GET", WS_ROUTES.burrow], + ["GET", WS_ROUTES.client], ], }; @@ -343,6 +349,7 @@ test("each bindings mapper passes only what its Worker uses", () => { HYPERDRIVE: { connectionString: "postgres://example.test/none" }, ASSETS: { fetch: async () => new Response() }, ONE_TIME_ROOM: {} as DurableObjectNamespace, + RELAY_ROOM: {} as DurableObjectNamespace, ONE_TIME_MINT_LIMIT: {} as RateLimit, ONE_TIME_JOIN_LIMIT: {} as RateLimit, RELAY_SIGNIN_LIMIT: {} as RateLimit, @@ -365,12 +372,14 @@ test("each bindings mapper passes only what its Worker uses", () => { "HYPERDRIVE", "POSTMARK_SERVER_TOKEN", "RELAY_APPROVE_LIMIT", + "RELAY_ROOM", ].sort(), ); expect(accountPreviewBindings(env)).toEqual({ APP_ORIGIN: env.APP_ORIGIN, ASSETS: env.ASSETS, RELAY_APPROVE_LIMIT: env.RELAY_APPROVE_LIMIT, + RELAY_ROOM: env.RELAY_ROOM, AUTH_SECRET: env.AUTH_SECRET, BUILD_SHA: sha, HYPERDRIVE: env.HYPERDRIVE, @@ -391,6 +400,7 @@ test("each bindings mapper passes only what its Worker uses", () => { "RELAY_ENROLL_BEGIN_LIMIT", "RELAY_ENROLL_POLL_LIMIT", "RELAY_ENROLL_SECRET", + "RELAY_ROOM", "RELAY_SETUP_LIMIT", "RELAY_SIGNIN_LIMIT", ].sort(), @@ -463,6 +473,9 @@ test("the cookie gate needs no login creation time; approval reads it and fails throw new Error("The limit is past the recent-login check"); }, } as RateLimit, + closeBurrow: async () => { + throw new Error("No approval removes a Burrow"); + }, }); const origin = "https://account.example.test"; // The voice-token routes' gate admits the admin whatever `createdAt` says. diff --git a/hosted/server/tests/bundle.ts b/hosted/server/tests/bundle.ts index fa6f76bfa..c6889ec24 100644 --- a/hosted/server/tests/bundle.ts +++ b/hosted/server/tests/bundle.ts @@ -5,11 +5,14 @@ import { builtinModules } from "node:module"; import { WORKERS, parseConfig } from "../../scripts/workers.mjs"; interface WranglerConfig { + name: string; main: string; vars: { APP_ORIGIN: string }; compatibility_date: string; compatibility_flags: string[]; - durable_objects?: { bindings: { name: string; class_name: string }[] }; + durable_objects?: { + bindings: { name: string; class_name: string; script_name?: string }[]; + }; migrations?: { tag: string; new_sqlite_classes?: string[] }[]; ratelimits?: { name: string; @@ -39,27 +42,38 @@ export const ENTRIES = each((config) => config.main); type V4Options = Parameters[0]; +/** Whether the Worker that implements `className` (`script`, or `config`'s own) made it SQLite-backed. */ +function sqliteClass(config: WranglerConfig, className: string, script?: string) { + const owner = script ? Object.values(wrangler).find((worker) => worker.name === script)! : config; + return !!owner.migrations?.some((migration) => + migration.new_sqlite_classes?.includes(className), + ); +} + /** - * Miniflare options running `script` as Worker `name`: compatibility from - * that Worker's own config, and its Durable Objects and rate limits bound as - * the config declares them. `extra` adds the rest, and wins. + * Miniflare options running `script` as Worker `name`, under its config's + * script name: compatibility from that Worker's own config, and its Durable + * Objects — another Worker's by that Worker's script name — and rate limits + * bound as the config declares them. `extra` adds the rest, and wins. A + * Worker binding another's class runs in one Miniflare beside it, real or + * {@link standIn}. */ export function miniflareOptions(name: Name, script: string, extra: Partial = {}) { const config = wrangler[name]; return convertV4MiniflareOptions({ + name: config.name, modules: true, script, compatibilityDate: config.compatibility_date, compatibilityFlags: config.compatibility_flags, ...(config.durable_objects && { durableObjects: Object.fromEntries( - config.durable_objects.bindings.map(({ name, class_name }) => [ + config.durable_objects.bindings.map(({ name, class_name, script_name }) => [ name, { className: class_name, - useSQLite: !!config.migrations?.some((migration) => - migration.new_sqlite_classes?.includes(class_name), - ), + ...(script_name && { scriptName: script_name }), + useSQLite: sqliteClass(config, class_name, script_name), }, ]), ), @@ -73,6 +87,42 @@ export function miniflareOptions(name: Name, script: string, extra: Partial !script_name) + .map(({ class_name }) => `export class ${class_name} extends DurableObject {}`); + return miniflareOptions( + name, + [ + `import { DurableObject } from "cloudflare:workers";`, + ...classes, + `export default { fetch: () => new Response(null, { status: 404 }) };`, + ].join("\n"), + ); +} + +type Options = ReturnType; + +/** One Miniflare running `main` — which `dispatchFetch` reaches unless a route says otherwise — and `siblings`. */ +export function together(main: Options, ...siblings: Options[]): Options { + return { ...main, workers: [...main.workers, ...siblings.flatMap((sibling) => sibling.workers)] }; +} + +/** `options` for Worker `name`, beside a stand-in for each Worker whose class its config binds by `script_name`. */ +export function alone(name: Name, options: Options): Options { + const scripts = new Set( + (wrangler[name].durable_objects?.bindings ?? []).flatMap(({ script_name }) => + script_name ? [script_name] : [], + ), + ); + return together(options, ...NAMES.filter((other) => scripts.has(wrangler[other].name)).map(standIn)); +} + /** A Worker entry bundled for workerd the way Wrangler bundles it. */ export function bundleWorker(entry: string, inject: string[] = []) { return build({ diff --git a/hosted/server/tests/one-time.test.ts b/hosted/server/tests/one-time.test.ts index 5dd2370f9..602350478 100644 --- a/hosted/server/tests/one-time.test.ts +++ b/hosted/server/tests/one-time.test.ts @@ -7,8 +7,8 @@ import { MAX_ONE_TIME_FRAME_LENGTH, ONE_TIME_LINK_TTL_MS, ONE_TIME_PAGE_PATH, - ONE_TIME_PING, - ONE_TIME_PONG, + RELAY_PING, + RELAY_PONG, ONE_TIME_ROOM_PARAM, ONE_TIME_WS_ROUTES, WS_CLOSE_ONE_TIME_DEADLINE, @@ -29,7 +29,7 @@ import * as smoke from "../../scripts/one-time-smoke.mjs"; import { pocketContentSecurityPolicy } from "remote-lib-common"; import { oneTimePagePolicy, relayRules, RUNS_NOTHING_POLICY } from "../headers"; import { rateLimitKey } from "../one-time"; -import { ENTRIES, ORIGINS, bundleWorker, miniflareOptions } from "./bundle"; +import { ENTRIES, ORIGINS, bundleWorker, miniflareOptions, wrangler } from "./bundle"; import { limitOf, untilLimited } from "./rate-limit"; import { TEST_ROOM_LIMITS } from "./one-time-limits"; import { rawUpgrade, type RawSocket } from "./raw-socket"; @@ -229,11 +229,11 @@ test("frames cross both ways byte for byte, JSON or not", async () => { test("pings are answered by the runtime, never forwarded or counted", async () => { const { burrow, phone } = await pair(); for (let i = 0; i < MAX_ONE_TIME_FORWARDED + 5; i++) { - phone.send(ONE_TIME_PING); - expect(await phone.next()).toBe(ONE_TIME_PONG); + phone.send(RELAY_PING); + expect(await phone.next()).toBe(RELAY_PONG); } - burrow.send(ONE_TIME_PING); - expect(await burrow.next()).toBe(ONE_TIME_PONG); + burrow.send(RELAY_PING); + expect(await burrow.next()).toBe(RELAY_PONG); for (let i = 0; i < MAX_ONE_TIME_FORWARDED; i++) { phone.send(`frame ${i}`); expect(await burrow.next()).toBe(`frame ${i}`); @@ -332,7 +332,7 @@ test("either end leaving closes the other and ends the room", async () => { test("a hibernated room keeps its join and its count", async () => { const { burrow, frame } = await mint(); const hibernate = () => - production.mf.unsafeEvictDurableObject("", "OneTimeRoom", { + production.mf.unsafeEvictDurableObject(wrangler.relay.name, "OneTimeRoom", { name: frame.roomId, webSockets: "hibernate", }); diff --git a/hosted/server/tests/pocket.test.ts b/hosted/server/tests/pocket.test.ts index 4040cf7ee..02aa93006 100644 --- a/hosted/server/tests/pocket.test.ts +++ b/hosted/server/tests/pocket.test.ts @@ -149,7 +149,9 @@ test("the API and socket routes are never Pocket's: no shell, no page policy, no // The self-host installers' probe is not Hosted's. expect((await get("/api/hello")).status).toBe(404); expect(await (await get(API_ROUTES.pushConfig)).json()).toEqual({ applicationServerKey: null }); - expect((await get(WS_ROUTES.client)).status).toBe(404); + // A socket route answers only an upgrade, and only after its Origin check. + expect((await get(WS_ROUTES.client)).status).toBe(403); + expect((await get(WS_ROUTES.burrow)).status).toBe(426); }); test("a percent-encoded /connect path is the page's route, under its policy and without the camera", async () => { diff --git a/hosted/server/tests/relay-room-entry.ts b/hosted/server/tests/relay-room-entry.ts new file mode 100644 index 000000000..bb07c23e9 --- /dev/null +++ b/hosted/server/tests/relay-room-entry.ts @@ -0,0 +1,130 @@ +import type { ExecutionContext } from "hono"; +import { WS_ROUTES } from "remote-lib-common"; +import { RELAY_ROOM_PARAMS } from "../relay-room-contract"; +import worker, { + OneTimeRoom, + RelayRoom as ProductionRoom, + RelayRows as ProductionRows, +} from "../relay-worker"; + +// Module state outlives an evicted object, so it counts what woke one. +let constructed = 0; +let handled = 0; +let skewMs = 0; +/** Whether a row read stalls, as a slow database would. */ +let rowsStalled = false; +/** What each upgrade handed the object: its header names and search parameters. */ +const upgrades: { headers: string[]; params: string[] }[] = []; + +/** + * The production `RelayRoom`, counting its wake-ups and handler calls, keeping + * what each upgrade carried, and on a clock a test can move. + */ +export class RelayRoom extends ProductionRoom { + constructor(ctx: DurableObjectState, env: unknown) { + super(ctx, env); + constructed += 1; + } + + protected override now() { + return Date.now() + skewMs; + } + + override async fetch(request: Request) { + upgrades.push({ + headers: [...request.headers.keys()], + params: [...new URL(request.url).searchParams.keys()], + }); + return super.fetch(request); + } + + override async webSocketMessage(ws: WorkerWebSocket, message: string | ArrayBuffer) { + handled += 1; + return super.webSocketMessage(ws, message); + } + + override async webSocketClose(ws: WorkerWebSocket, code: number) { + handled += 1; + return super.webSocketClose(ws, code); + } + + override async alarm() { + handled += 1; + return super.alarm(); + } + + /** The counters, the upgrades so far, every socket's attachment, what storage holds, and the alarm. */ + async probe() { + return { + constructed, + handled, + upgrades, + attachments: this.ctx.getWebSockets().map((ws) => ws.deserializeAttachment()), + storage: Object.fromEntries(await this.ctx.storage.list()), + alarm: await this.ctx.storage.getAlarm(), + }; + } + + /** Move this object's clock by `ms` from real time. */ + async skew(ms: number) { + skewMs = ms; + } + + /** Run the alarm now, as its time arriving would. */ + async fire() { + await this.alarm(); + } + + /** Stall every row read from now, or stop stalling new ones. */ + async stallRows(on: boolean) { + rowsStalled = on; + } +} + +/** + * The production `RelayRows`, whose read, while stalled, waits past the 30 s + * the runtime gives a `blockConcurrencyWhile` callback; a pending timer keeps + * workerd from cancelling it as a hung request. + */ +export class RelayRows extends ProductionRows { + override async burrows(burrowIds: string[]) { + if (rowsStalled) await new Promise((resolve) => setTimeout(resolve, 60_000)); + return super.burrows(burrowIds); + } +} + +export { OneTimeRoom }; + +type TestEnv = { RELAY_ROOM: DurableObjectNamespace unknown>> }; + +/** + * The relay Worker, plus `POST /__test/room/?account=`: that + * RPC on the account's object with the JSON body as its arguments, from + * inside workerd so no stub outlives the request. `forge` instead hands the + * object a Burrow upgrade naming the account and the Burrow in the body, as + * the Worker would after resolving a token, and answers its status; a socket + * it opens is closed at once. + */ +export default { + async fetch(request: Request, env: TestEnv, ctx: ExecutionContext) { + const url = new URL(request.url); + const method = /^\/__test\/room\/(\w+)$/.exec(url.pathname)?.[1]; + if (!method) return worker.fetch(request, env as never, ctx); + const account = url.searchParams.get("account")!; + const room = env.RELAY_ROOM.get(env.RELAY_ROOM.idFromName(account)); + const args = (await request.json()) as unknown[]; + if (method === "forge") { + const target = new URL(WS_ROUTES.burrow, url.origin); + target.searchParams.set(RELAY_ROOM_PARAMS.account, String(args[0])); + target.searchParams.set(RELAY_ROOM_PARAMS.burrowId, String(args[1])); + const response = await room.fetch(new Request(target, { headers: { upgrade: "websocket" } })); + const socket = (response as { webSocket?: WorkerWebSocket | null }).webSocket; + if (socket) { + socket.accept(); + socket.close(1000); + } + return Response.json(response.status); + } + return Response.json((await room[method](...args)) ?? null); + }, +}; diff --git a/hosted/server/tests/relay-room.test.ts b/hosted/server/tests/relay-room.test.ts new file mode 100644 index 000000000..bbd753ea8 --- /dev/null +++ b/hosted/server/tests/relay-room.test.ts @@ -0,0 +1,718 @@ +import { test, expect, beforeAll, afterAll, vi } from "vitest"; +import { randomUUID } from "node:crypto"; +import { digest } from "@pgstencil/auth/security"; +import { Hono } from "hono"; +import { Miniflare, Response as WorkerResponse } from "miniflare"; +import { createTestContext } from "pgstencil/testing"; +import { queryDatabase } from "pgstencil/postgres"; +import { + API_ROUTES, + MAX_RELAY_CLIENT_SOCKETS, + NOT_ENTITLED_ERROR, + RELAY_IDLE_TIMEOUT_MS, + RELAY_PING, + RELAY_PONG, + UNAUTHORIZED_ERROR, + UNKNOWN_BURROW_TOKEN_ERROR, + WS_CLOSE_BURROW_REPLACED, + WS_CLOSE_BURROW_REVOKED, + WS_CLOSE_IDLE, + WS_CLOSE_TRY_AGAIN_LATER, + WS_CLOSE_UNAUTHORIZED, + WS_CLOSE_UNAUTHORIZED_REASON, + WS_ROUTES, + WS_TOKEN_PARAM, + generateNoiseKeyPair, +} from "remote-lib-common"; +import { + SimAuthenticator, + randomRoutingId, + randomSecret, + registrationClientData, +} from "../../../remote-lib-common/test/harness/actors.mjs"; +import { e2eClientFrame, newE2eId } from "../../../remote-lib-common/test/harness/envelope.mjs"; +import { FakeBurrow } from "../../../remote-lib-common/test/harness/fake-burrow.mjs"; +import { FakeClient } from "../../../remote-lib-common/test/harness/fake-client.mjs"; +import { openFrameSocket, until } from "../../../remote-lib-common/test/harness/frame-socket.mjs"; +import { e2eCases, socketCases } from "../../../remote-lib-common/test/harness/relay-parity.mjs"; +import { ADMIN_EMAIL } from "../admin"; +import { migrations } from "../migrations"; +import { relayAccountRoutes } from "../relay-account"; +import { RELAY_ROOM_SWEEP_MS, RELAY_ROW_READ_TIMEOUT_MS } from "../relay-room-contract"; +import { ORIGINS, TEST_ENROLL_SECRET, bundleWorker, miniflareOptions, wrangler } from "./bundle"; + +// The Hosted Relay's sockets and its per-account `RelayRoom` +// (`docs/specs/hosted.md` -> "Relay sockets") in real workerd against real +// Postgres: the routing every Relay shares (`relay-parity.mjs`), driven as on +// the self-host Relay, then what only the Durable Object has — hibernation, +// one object per account, the revocation RPC, the session alarm, and the +// liveness the runtime answers. + +const origin = ORIGINS.relay; +const rpId = new URL(origin).hostname; +/** A browser's Origin on a Client socket: Pocket is same-origin. */ +const POCKET = { headers: { origin } }; + +type Authenticator = Awaited>; +const newAuthenticator = ( + SimAuthenticator.create as unknown as (options: { + rpId: string; + userVerification?: boolean; + }) => Promise +).bind(SimAuthenticator, { rpId, userVerification: false }); + +/** A harness peer, typed by hand: its JavaScript options are inferred from the first caller. */ +type Peer = { ready: Promise; close(): void } & Record; +const harness = (Class: unknown, options: Record) => + new (Class as new (options: Record) => Peer)(options); + +/** What `probe()` on the test entry's `RelayRoom` answers. */ +interface Probe { + constructed: number; + handled: number; + upgrades: { headers: string[]; params: string[] }[]; + attachments: Record[]; + storage: Record; + alarm: number | null; +} +/** RPC on the account's object, through the test entry's `/__test/room/` route. */ +async function rpc(account: string, method: string, ...args: unknown[]): Promise { + const response = await relay.dispatchFetch( + `${origin}/__test/room/${method}?account=${encodeURIComponent(account)}`, + { method: "POST", body: JSON.stringify(args) }, + ); + expect(response.status, method).toBe(200); + return (await response.json()) as T; +} +/** `account`'s object, as the Workers reach it. */ +const roomOf = (account: string) => ({ + closeBurrow: (claimed: string, burrowId: string) => rpc(account, "closeBurrow", claimed, burrowId), + onlineBurrows: (claimed: string) => rpc(account, "onlineBurrows", claimed), + probe: () => rpc(account, "probe"), + skew: (ms: number) => rpc(account, "skew", ms), + /** Run the alarm now. */ + fire: () => rpc(account, "fire"), + /** Stall every row read the object makes, or stop. */ + stallRows: (on: boolean) => rpc(account, "stallRows", on), + /** The status of an upgrade naming `claimed` and `burrowId`. */ + forge: (claimed: string, burrowId: string) => rpc(account, "forge", claimed, burrowId), +}); + +let context: Awaited>; +let relay: Miniflare; +/** Miniflare's own address; `upstream` makes every request to it one for `origin`. */ +let base: URL; +let wsBase: string; + +beforeAll(async () => { + context = await createTestContext({ migrations }); + relay = new Miniflare({ + ...miniflareOptions( + "relay", + (await bundleWorker("server/tests/relay-room-entry.ts")).outputFiles[0].text, + { + bindings: { + APP_ORIGIN: origin, + ACCOUNT_ORIGIN: ORIGINS.account, + RELAY_ENROLL_SECRET: TEST_ENROLL_SECRET, + }, + hyperdrives: { HYPERDRIVE: context.database.url }, + serviceBindings: { + ASSETS: () => + new WorkerResponse("", { headers: { "content-type": "text/html" } }), + }, + }, + ), + upstream: origin, + }); + base = await relay.ready; + wsBase = base.href.replace(/^http/, "ws").replace(/\/$/, ""); +}); +afterAll(async () => { + await relay?.dispose(); + await context?.close(); +}); + +const sql = = Record>( + text: string, + values: unknown[] = [], +) => queryDatabase(context.database.url, text, values); + +let addresses = 0; +/** One API request, each from its own address so no rate limit is shared. */ +async function call( + path: string, + { body, bearer, method = "POST" }: { body?: unknown; bearer?: string; method?: string } = {}, +) { + const response = await relay.dispatchFetch(origin + path, { + method, + headers: { + "cf-connecting-ip": `198.51.100.${++addresses % 250}`, + "content-type": "application/json", + ...(bearer ? { authorization: `Bearer ${bearer}` } : {}), + }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + }); + const text = await response.text(); + return { status: response.status, json: (text ? JSON.parse(text) : null) as Record }; +} + +/** A user row made the one entitled account: `ADMIN_EMAIL` is unique, so whoever held it moves. */ +async function entitled() { + const id = randomUUID(); + await sql(`UPDATE "user" SET email = id || '@example.test' WHERE email = $1`, [ADMIN_EMAIL]); + await sql(`INSERT INTO "user" (id, name, email, "emailVerified") VALUES ($1, $1, $2, true)`, [ + id, + ADMIN_EMAIL, + ]); + return id; +} + +/** An enrolled Burrow row, as the device-code poll writes one. */ +async function burrowRow(userId: string) { + const burrowId = randomRoutingId() as string; + const burrowToken = randomSecret() as string; + await sql( + `INSERT INTO dormouse_relay_burrows ("burrowId", "userId", "tokenHash") VALUES ($1, $2, $3)`, + [burrowId, userId, digest(burrowToken)], + ); + return { burrowId, burrowToken }; +} + +/** + * A passkey registered off `burrowToken`'s setup code and signed in, through + * the Hosted routes as Pocket drives them. + */ +async function signedIn(burrowToken: string) { + const authenticator = await newAuthenticator(); + const minted = await call(API_ROUTES.burrowSetupToken, { bearer: burrowToken }); + const setupToken = minted.json.token as string; + const begun = await call(API_ROUTES.setupBegin, { body: { setupToken } }); + const finished = await call(API_ROUTES.setupFinish, { + body: { + setupToken, + credentialId: authenticator.credentialId, + publicKey: authenticator.publicKey, + clientDataJSON: registrationClientData({ challenge: begun.json.challenge, origin }), + label: "Phone", + }, + }); + expect(finished.status).toBe(200); + const challenge = (await call(API_ROUTES.signinBegin, { body: {} })).json.challenge; + const assertion = await authenticator.assert({ challenge, origin }); + const signed = await call(API_ROUTES.signinFinish, { body: { assertion } }); + expect(signed.status).toBe(200); + return { authenticator, sessionToken: signed.json.sessionToken as string }; +} + +const burrowUrl = (token: string) => `${wsBase}${WS_ROUTES.burrow}?${WS_TOKEN_PARAM}=${token}`; +const clientUrl = (token: string) => `${wsBase}${WS_ROUTES.client}?${WS_TOKEN_PARAM}=${token}`; + +type FrameSocket = ReturnType; +const opened = new Set(); +function socket(url: string, init?: { headers: Record }) { + const ws = openFrameSocket(url, init); + opened.add(ws); + return ws; +} +async function open(url: string, init?: { headers: Record }) { + const ws = socket(url, init); + await ws.ready; + return ws; +} +function closeAll() { + for (const ws of opened) ws.close(); + opened.clear(); +} + +/** An entitled account, signed in, with the parity driver over its sockets. */ +async function account() { + const userId = await entitled(); + const first = await burrowRow(userId); + const { sessionToken, authenticator } = await signedIn(first.burrowToken); + const openClient = () => socket(clientUrl(sessionToken), POCKET); + return { + userId, + sessionToken, + authenticator, + /** This account's object. */ + room: async () => roomOf(userId), + driver: { + async connectBurrow() { + const row = await burrowRow(userId); + return { ...row, socket: await open(burrowUrl(row.burrowToken)) }; + }, + reconnectBurrow: (burrowToken: string) => open(burrowUrl(burrowToken)), + openClient, + async connectClient() { + const ws = openClient(); + await ws.ready; + return ws; + }, + }, + }; +} + +/** Whether an attachment is a socket the object still routes. */ +const live = (conn: Record) => !conn.retired; + +/** Evict `account`'s object, its sockets hibernated, as the runtime does when nothing is happening. */ +const hibernate = (account: string) => + relay.unsafeEvictDurableObject(wrangler.relay.name, "RelayRoom", { + name: account, + webSockets: "hibernate", + }); + +for (const { name, run } of socketCases) + test(`parity: ${name}`, async ({ onTestFinished }) => { + onTestFinished(closeAll); + await run((await account()).driver); + }); + +for (const { name, run } of e2eCases) + test(`parity: ${name}`, async ({ onTestFinished }) => { + const owner = await account(); + const burrowStatic = await generateNoiseKeyPair(); + const clientStatic = await generateNoiseKeyPair(); + const peers: { close(): void }[] = []; + onTestFinished(() => { + for (const peer of peers) peer.close(); + }); + const fakeBurrow = async ({ burrowId, burrowToken }: { burrowId: string; burrowToken: string }) => { + const peer = harness(FakeBurrow, { + relayUrl: wsBase, + burrowToken, + burrowId, + origin, + rpId, + noiseStaticKeyPair: burrowStatic, + }); + peers.push(peer); + await peer.ready; + return peer; + }; + const enrollment = await burrowRow(owner.userId); + const burrow = await fakeBurrow(enrollment); + const client = harness(FakeClient, { + relayUrl: base.href.replace(/\/$/, ""), + sessionToken: owner.sessionToken, + burrowId: enrollment.burrowId, + staticKeyPair: clientStatic, + burrowStaticPublicKey: burrowStatic.publicKey, + origin, + rpId, + socketInit: POCKET, + }); + peers.push(client); + await client.ready; + await run({ + burrow, + client, + authenticator: owner.authenticator, + accountId: owner.userId, + enrollment, + burrowStatic, + clientStatic, + replacementBurrow: () => fakeBurrow(enrollment), + secondBurrow: async () => fakeBurrow(await burrowRow(owner.userId)), + close: async () => {}, + }); + }); + +test("each upgrade resolves its token and Origin first, and hands the object only what they resolved to", async ({ + onTestFinished, +}) => { + onTestFinished(closeAll); + const owner = await account(); + const { burrowId, burrowToken } = await burrowRow(owner.userId); + + /** An upgrade's refusal: its status and body. */ + const refusal = async (url: string, headers: Record = {}) => { + const response = await relay.dispatchFetch(url.replace(wsBase, origin), { + headers: { upgrade: "websocket", ...headers }, + }); + return { status: response.status, json: await response.json() }; + }; + // A Burrow socket carries no Origin: a browser page always sends one. + expect(await refusal(burrowUrl(burrowToken), POCKET.headers)).toEqual({ + status: 403, + json: { error: "forbidden" }, + }); + for (const token of ["", "short", randomSecret()]) + expect(await refusal(`${origin}${WS_ROUTES.burrow}?${WS_TOKEN_PARAM}=${token}`), token).toEqual({ + status: 401, + json: { error: UNKNOWN_BURROW_TOKEN_ERROR }, + }); + // A Client socket carries exactly this origin: no other page's, and not none. + for (const headers of [{}, { origin: ORIGINS.account }, { origin: `${origin}.evil.test` }] as Record< + string, + string + >[]) + expect(await refusal(clientUrl(owner.sessionToken), headers)).toEqual({ + status: 403, + json: { error: "forbidden" }, + }); + for (const token of ["", "short", randomSecret()]) + expect(await refusal(clientUrl(token), POCKET.headers), token).toEqual({ + status: 401, + json: { error: UNAUTHORIZED_ERROR }, + }); + + // Admitted, the object hears the account and the Burrow or the expiry, and + // not one of the caller's headers or its token. + const before = (await (await owner.room()).probe()).upgrades.length; + await open(burrowUrl(burrowToken), { + headers: { cookie: "a=b", authorization: "Bearer x", "cf-connecting-ip": "203.0.113.9" }, + }); + await open(clientUrl(owner.sessionToken), { headers: { ...POCKET.headers, cookie: "a=b" } }); + const { upgrades, storage } = await (await owner.room()).probe(); + expect(upgrades.slice(before)).toEqual([ + { headers: ["upgrade"], params: ["account", "burrow"] }, + { headers: ["upgrade"], params: ["account", "expires"] }, + ]); + // Its one durable value is the account it serves. + expect(storage).toEqual({ account: owner.userId }); + expect((await call(API_ROUTES.burrows, { bearer: owner.sessionToken, method: "GET" })).json.burrows).toContainEqual({ + burrowId, + online: true, + }); + + // A de-entitled owner opens neither socket; Pocket reads the 401 as expiry. + await entitled(); + expect(await refusal(burrowUrl(burrowToken))).toEqual({ + status: 403, + json: { error: NOT_ENTITLED_ERROR }, + }); + expect(await refusal(clientUrl(owner.sessionToken), POCKET.headers)).toEqual({ + status: 401, + json: { error: UNAUTHORIZED_ERROR }, + }); +}); + +test("routing survives hibernation: across a binding, a replacement, and a pending session alarm", async ({ + onTestFinished, +}) => { + onTestFinished(closeAll); + const owner = await account(); + const first = await owner.driver.connectBurrow(); + const client = await owner.driver.connectClient(); + const init = e2eClientFrame(first.burrowId); + client.send(init); + const forwarded = await first.socket.take(); + + // Between init and transport: the binding and the clientId are the attachment's. + await hibernate(owner.userId); + client.send(e2eClientFrame(first.burrowId, { step: "transport", id: init.id })); + const transport = await first.socket.take(); + expect(transport).toMatchObject({ step: "transport", clientId: forwarded.clientId }); + first.socket.send({ ...transport, step: "transport", burrowId: undefined, ct: "YmFy" }); + expect(await client.take()).toMatchObject({ t: "e2e", burrowId: first.burrowId, ct: "YmFy" }); + + // Between replacement steps. + await hibernate(owner.userId); + const second = await owner.driver.reconnectBurrow(first.burrowToken); + expect((await first.socket.closed).code).toBe(WS_CLOSE_BURROW_REPLACED); + expect(await client.take()).toEqual({ t: "burrow-gone" }); + await hibernate(owner.userId); + client.send(e2eClientFrame(first.burrowId, { step: "transport" })); + expect(await second.quiet(150)).toBe(true); + await hibernate(owner.userId); + client.send(e2eClientFrame(first.burrowId)); + expect(await second.take()).toMatchObject({ step: "init", clientId: forwarded.clientId }); + + // A session about to expire: its alarm wakes the object it was armed in. + await sql(`UPDATE dormouse_relay_sessions SET "expiresAt" = now() + interval '2 seconds'`); + const expiring = await owner.driver.connectClient(); + expiring.send(e2eClientFrame(first.burrowId)); + const bound = await second.take(); + await hibernate(owner.userId); + const closed = await expiring.closed; + expect([closed.code, closed.reason]).toEqual([WS_CLOSE_UNAUTHORIZED, WS_CLOSE_UNAUTHORIZED_REASON]); + expect(await second.take(3000)).toEqual({ t: "client-gone", clientId: bound.clientId }); + // The Client opened before the update holds its own expiry: still routed. + client.send(e2eClientFrame(first.burrowId, { step: "transport" })); + expect(await second.take()).toMatchObject({ step: "transport" }); +}); + +test("the alarm is the earliest Client expiry or Burrow sweep; a close leaves it, and the alarm re-arms or clears", async ({ + onTestFinished, +}) => { + onTestFinished(closeAll); + const owner = await account(); + const room = await owner.room(); + const expiry = async () => + ( + await sql<{ expiresAt: number }>( + `SELECT floor(extract(epoch from "expiresAt") * 1000)::float8 AS "expiresAt" + FROM dormouse_relay_sessions WHERE "tokenHash" = $1`, + [digest(owner.sessionToken)], + ) + )[0]!.expiresAt; + const later = await owner.driver.connectClient(); + expect((await room.probe()).alarm).toBe(await expiry()); + await sql(`UPDATE dormouse_relay_sessions SET "expiresAt" = "expiresAt" - interval '1 hour'`); + const sooner = await owner.driver.connectClient(); + const soonest = await expiry(); + expect((await room.probe()).alarm).toBe(soonest); + // A close writes nothing; the alarm, when it comes, moves to what is left. + sooner.close(); + await sooner.closed; + await until(async () => (await room.probe()).attachments.filter(live).length === 1); + expect((await room.probe()).alarm).toBe(soonest); + await room.fire(); + expect((await room.probe()).alarm).toBe(soonest + 3_600_000); + + // A Burrow socket brings the alarm to its sweep, at most an hour off. + const before = Date.now(); + await owner.driver.connectBurrow(); + const swept = (await room.probe()).alarm!; + expect(swept).toBeGreaterThanOrEqual(before + RELAY_ROOM_SWEEP_MS); + expect(swept).toBeLessThanOrEqual(Date.now() + RELAY_ROOM_SWEEP_MS); + // And a sweep, while the Burrow is held, schedules the next one an hour on. + const sweeping = Date.now(); + await room.fire(); + const next = (await room.probe()).alarm!; + expect(next).toBeGreaterThanOrEqual(sweeping + RELAY_ROOM_SWEEP_MS); + expect(next).toBeLessThanOrEqual(Date.now() + RELAY_ROOM_SWEEP_MS); + closeAll(); + await until(async () => (await room.probe()).attachments.filter(live).length === 0); + await room.fire(); + expect((await room.probe()).alarm).toBe(null); +}); + +test("the sweep closes a Burrow removed or de-entitled behind its socket's back, and keeps the rest", async ({ + onTestFinished, +}) => { + onTestFinished(closeAll); + const owner = await account(); + const room = await owner.room(); + const removed = await owner.driver.connectBurrow(); + const kept = await owner.driver.connectBurrow(); + const client = await owner.driver.connectClient(); + client.send(e2eClientFrame(removed.burrowId)); + await removed.socket.take(); + + // Removed with no RPC, as a removal whose close never arrived. + await sql(`DELETE FROM dormouse_relay_burrows WHERE "burrowId" = $1`, [removed.burrowId]); + await room.fire(); + expect((await removed.socket.closed).code).toBe(WS_CLOSE_BURROW_REVOKED); + expect(await client.take()).toEqual({ t: "burrow-gone" }); + expect(await room.onlineBurrows(owner.userId)).toEqual([kept.burrowId]); + expect(await kept.socket.quiet(150)).toBe(true); + + // The owner loses the entitlement: every Burrow socket it holds goes. + await entitled(); + await room.fire(); + expect((await kept.socket.closed).code).toBe(WS_CLOSE_BURROW_REVOKED); + expect(await room.onlineBurrows(owner.userId)).toEqual([]); +}); + +test("a Burrow removed between the Worker's token check and the object's accept is refused", async ({ + onTestFinished, +}) => { + onTestFinished(closeAll); + const owner = await account(); + const room = await owner.room(); + // The object holds the account already: the refusals below are the row's. + await owner.driver.connectClient(); + const enrolled = await burrowRow(owner.userId); + // The upgrade as the Worker hands it on once the token resolved: accepted. + expect(await room.forge(owner.userId, enrolled.burrowId)).toBe(101); + const removed = await burrowRow(owner.userId); + await sql(`DELETE FROM dormouse_relay_burrows WHERE "burrowId" = $1`, [removed.burrowId]); + expect(await room.forge(owner.userId, removed.burrowId)).toBe(401); + // Another account's Burrow, and one whose owner lost the entitlement. + const other = await account(); + expect(await room.forge(owner.userId, (await burrowRow(other.userId)).burrowId)).toBe(401); + expect(await room.forge(owner.userId, (await burrowRow(owner.userId)).burrowId)).toBe(403); +}); + +test("a stalled row read answers the upgrade 503 within its bound and leaves the sweep for later, the object's sockets open", async ({ + onTestFinished, +}) => { + const owner = await account(); + const room = await owner.room(); + onTestFinished(async () => { + closeAll(); + await room.stallRows(false); + }); + const held = await owner.driver.connectBurrow(); + const client = await owner.driver.connectClient(); + const enrolled = await burrowRow(owner.userId); + await room.stallRows(true); + + // Unbounded, the read would hold `blockConcurrencyWhile` past the runtime's + // 30 s and reset the object, every socket with it. + const started = Date.now(); + expect(await room.forge(owner.userId, enrolled.burrowId)).toBe(503); + expect(Date.now() - started).toBeLessThan(RELAY_ROW_READ_TIMEOUT_MS + 5_000); + // The sweep's read fails the same way and closes nothing. + await room.fire(); + expect(await held.socket.quiet(150)).toBe(true); + + client.send(e2eClientFrame(held.burrowId)); + expect(await held.socket.take()).toMatchObject({ step: "init" }); + expect(await room.onlineBurrows(owner.userId)).toEqual([held.burrowId]); + await room.stallRows(false); + expect(await room.forge(owner.userId, enrolled.burrowId)).toBe(101); +}); + +test("a removal answers 204 once the row is gone, even when closing its socket fails", async () => { + const userId = await entitled(); + const { burrowId } = await burrowRow(userId); + const logged = vi.spyOn(console, "error").mockImplementation(() => {}); + const app = new Hono(); + relayAccountRoutes(app, () => ({ + databaseUrl: context.database.url, + auth: async () => + Response.json({ user: { id: userId, email: ADMIN_EMAIL, emailVerified: true }, session: {} }), + approveLimit: {} as RateLimit, + closeBurrow: async () => { + throw new Error("the relay is unreachable"); + }, + })); + const response = await app.request(`${ORIGINS.account}/api/relay/burrows/${burrowId}`, { + method: "DELETE", + headers: { origin: ORIGINS.account }, + }); + expect(response.status).toBe(204); + expect(await sql(`SELECT 1 FROM dormouse_relay_burrows WHERE "burrowId" = $1`, [burrowId])).toEqual([]); + expect(logged).toHaveBeenCalledOnce(); + logged.mockRestore(); +}); + +test("a ping is answered without waking the object, and changes nothing it holds", async ({ + onTestFinished, +}) => { + onTestFinished(closeAll); + const owner = await account(); + const { burrowId, socket: burrow } = await owner.driver.connectBurrow(); + const client = await owner.driver.connectClient(); + client.send(e2eClientFrame(burrowId)); + await burrow.take(); + + const room = await owner.room(); + const before = await room.probe(); + await hibernate(owner.userId); + for (const ws of [client, burrow, client]) { + ws.ws.send(RELAY_PING); + expect(await ws.take()).toBe(RELAY_PONG); + } + const after = await room.probe(); + // The probe itself is the one wake-up; no handler ran for the pings. + expect(after.constructed).toBe(before.constructed + 1); + expect(after.handled).toBe(before.handled); + expect(after.attachments).toEqual(before.attachments); + expect(after.storage).toEqual(before.storage); +}); + +test("a Burrow silent past three ping intervals is gone when routed to, and is not reported online", async ({ + onTestFinished, +}) => { + onTestFinished(async () => { + closeAll(); + await (await owner.room()).skew(0); + }); + const owner = await account(); + const quiet = await owner.driver.connectBurrow(); + const pinging = await owner.driver.connectBurrow(); + const neverPinged = await owner.driver.connectBurrow(); + for (const { socket } of [quiet, pinging]) { + socket.ws.send(RELAY_PING); + expect(await socket.take()).toBe(RELAY_PONG); + } + const room = await owner.room(); + await room.skew(RELAY_IDLE_TIMEOUT_MS + 1_000); + // `pinging` pings again on time; `quiet` does not; `neverPinged` is never judged. + pinging.socket.ws.send(RELAY_PING); + expect(await pinging.socket.take()).toBe(RELAY_PONG); + + const client = await owner.driver.connectClient(); + client.send(e2eClientFrame(quiet.burrowId)); + expect((await client.take()).error).toBe(`burrow ${quiet.burrowId} is offline`); + expect((await quiet.socket.closed).code).toBe(WS_CLOSE_IDLE); + + // `pinging`'s last pong is now behind the skewed clock too: not online, and gone. + await room.skew(2 * RELAY_IDLE_TIMEOUT_MS); + expect(await room.onlineBurrows(owner.userId)).toEqual([neverPinged.burrowId]); + expect((await pinging.socket.closed).code).toBe(WS_CLOSE_IDLE); +}); + +test("at the Client cap, a silent socket is gone and a live one is never evicted", async ({ + onTestFinished, +}) => { + const owner = await account(); + const room = await owner.room(); + onTestFinished(async () => { + closeAll(); + await room.skew(0); + }); + const clients = []; + for (let i = 0; i < MAX_RELAY_CLIENT_SOCKETS; i += 1) clients.push(await owner.driver.connectClient()); + const [silent] = clients; + silent.ws.send(RELAY_PING); + expect(await silent.take()).toBe(RELAY_PONG); + // Full, and nobody silent yet: refused. + expect((await owner.driver.openClient().closed).code).toBe(WS_CLOSE_TRY_AGAIN_LATER); + await room.skew(RELAY_IDLE_TIMEOUT_MS + 1_000); + // The one that pinged and fell silent goes; those that never pinged stay. + const admitted = owner.driver.openClient(); + await admitted.ready; + expect((await silent.closed).code).toBe(WS_CLOSE_IDLE); + expect(await admitted.quiet(150)).toBe(true); + expect(clients.slice(1).every(({ ws }) => ws.readyState === WebSocket.OPEN)).toBe(true); +}); + +test("revocation closes a Burrow 4001 and tells its Clients; removal on the account pushes it", async ({ + onTestFinished, +}) => { + onTestFinished(closeAll); + const owner = await account(); + const { burrowId, socket: burrow } = await owner.driver.connectBurrow(); + const client = await owner.driver.connectClient(); + client.send(e2eClientFrame(burrowId)); + await burrow.take(); + const room = await owner.room(); + expect(await room.closeBurrow(owner.userId, newE2eId())).toBe(false); + await hibernate(owner.userId); + expect(await room.closeBurrow(owner.userId, burrowId)).toBe(true); + expect((await burrow.closed).code).toBe(WS_CLOSE_BURROW_REVOKED); + expect(await client.take()).toEqual({ t: "burrow-gone" }); + expect(await room.onlineBurrows(owner.userId)).toEqual([]); + expect(await room.closeBurrow(owner.userId, burrowId)).toBe(false); +}); + +test("one object per account: another account's session never reaches this account's Burrow", async ({ + onTestFinished, +}) => { + onTestFinished(closeAll); + const a = await account(); + const { burrowId, socket: burrowA } = await a.driver.connectBurrow(); + const clientA = await a.driver.connectClient(); + clientA.send(e2eClientFrame(burrowId)); + const { clientId } = await burrowA.take(); + + // B is entitled now; A's live sockets stay as they are. + const b = await account(); + const clientB = await b.driver.connectClient(); + for (const step of ["init", "transport"]) { + clientB.send(e2eClientFrame(burrowId, { step })); + if (step === "init") expect((await clientB.take()).error).toBe(`burrow ${burrowId} is offline`); + } + expect(await burrowA.quiet(150)).toBe(true); + expect((await call(API_ROUTES.burrows, { bearer: b.sessionToken, method: "GET" })).json.burrows).not.toContainEqual( + expect.objectContaining({ burrowId }), + ); + // A frame A's Burrow addresses to its own Client still reaches only that Client. + burrowA.send({ t: "e2e", clientId, kind: "pairing", id: newE2eId(), step: "response", ct: "YmFy" }); + expect((await clientA.take()).burrowId).toBe(burrowId); + expect(await clientB.quiet(150)).toBe(true); + + // A's object, handed B's account, refuses: no socket, no RPC, nothing written. + const roomA = await a.room(); + expect(await roomA.forge(b.userId, newE2eId())).toBe(403); + expect(await roomA.closeBurrow(b.userId, burrowId)).toBe(false); + expect(await roomA.onlineBurrows(b.userId)).toEqual([]); + expect((await roomA.probe()).storage).toEqual({ account: a.userId }); + expect(await roomA.onlineBurrows(a.userId)).toEqual([burrowId]); +}); diff --git a/hosted/server/tests/workers.test.ts b/hosted/server/tests/workers.test.ts index 30e40ea4d..cf4a489f8 100644 --- a/hosted/server/tests/workers.test.ts +++ b/hosted/server/tests/workers.test.ts @@ -22,13 +22,22 @@ import type { Session } from "../../src/api"; import { ADMIN_EMAIL } from "../admin"; import { CRON_SWEEP_CAP, SPEECH_SWEEP_CAP, VOICE_DAILY_CAP } from "../voice"; import { ALREADY_APPROVED, RECENT_LOGIN_REQUIRED } from "../relay-account"; -import { API_ROUTES, NOT_ENTITLED_ERROR } from "remote-lib-common"; +import { + API_ROUTES, + NOT_ENTITLED_ERROR, + WS_CLOSE_BURROW_REVOKED, + WS_ROUTES, + WS_TOKEN_PARAM, +} from "remote-lib-common"; +import { rawUpgrade } from "./raw-socket"; import { ENTRIES, ORIGINS, TEST_ENROLL_SECRET, bundleWorker, miniflareOptions, + together, + wrangler, type Name, } from "./bundle"; import { limitOf, untilLimited } from "./rate-limit"; @@ -80,6 +89,7 @@ const workerOptions = ( assets?: Handler; bindings: Record; outboundService: Handler; + routes?: string[]; }, ) => miniflareOptions(name, script, { @@ -179,32 +189,49 @@ async function fixture( }, }); }; + // The relay Worker runs beside the account, as its `RelayRoom` binding needs, + // on the same database, answering for its own host; its enrollment links + // name this account. const worker = new Miniflare( - workerOptions("account", { - script: ( - await (production === "preview" - ? previewBundle - : production - ? productionBundle - : testBundle) - ).outputFiles![0].text, - bindings, - database: context.database.url, - // Vite's content-hashed build output, with the SPA fallback answering - // every other path — including an unknown one under /assets/ — with the shell. - assets: (request) => - new URL(request.url).pathname === "/assets/app-abc123.js" - ? new WorkerResponse("export const build = 1;\n", { - headers: { "content-type": "text/javascript" }, - }) - : new WorkerResponse( - "Dormouse Hosted", - { - headers: { "content-type": "text/html" }, - }, - ), - outboundService, - }), + together( + workerOptions("account", { + script: ( + await (production === "preview" + ? previewBundle + : production + ? productionBundle + : testBundle) + ).outputFiles![0].text, + bindings, + database: context.database.url, + // Vite's content-hashed build output, with the SPA fallback answering + // every other path — including an unknown one under /assets/ — with the shell. + assets: (request) => + new URL(request.url).pathname === "/assets/app-abc123.js" + ? new WorkerResponse("export const build = 1;\n", { + headers: { "content-type": "text/javascript" }, + }) + : new WorkerResponse( + "Dormouse Hosted", + { + headers: { "content-type": "text/html" }, + }, + ), + outboundService, + }), + workerOptions("relay", { + script: (await relayBundle).outputFiles![0].text, + bindings: { + APP_ORIGIN: ORIGINS.relay, + ACCOUNT_ORIGIN: origin, + RELAY_ENROLL_SECRET: TEST_ENROLL_SECRET, + }, + database: context.database.url, + assets: () => new WorkerResponse("", { headers: { "content-type": "text/html" } }), + outboundService, + routes: [`${new URL(ORIGINS.relay).host}/*`], + }), + ), ); // The voice Worker on the same database, with every binding the account // has (its mapper drops what it does not use), started on first use. @@ -230,27 +257,7 @@ async function fixture( await started.ready; return started; })()); - // The relay Worker on the same database, its enrollment links naming this - // account, started on first use. - let relayWorker: Promise | undefined; - const relay = () => - (relayWorker ??= (async () => { - const started = new Miniflare( - workerOptions("relay", { - script: (await relayBundle).outputFiles![0].text, - bindings: { - APP_ORIGIN: ORIGINS.relay, - ACCOUNT_ORIGIN: origin, - RELAY_ENROLL_SECRET: TEST_ENROLL_SECRET, - }, - database: context.database.url, - assets: () => new WorkerResponse("", { headers: { "content-type": "text/html" } }), - outboundService, - }), - ); - await started.ready; - return started; - })()); + const relay = async () => worker; /** One relay request as a Burrow sends it: JSON, no cookie, no Origin. */ const burrowCall = async (path: string, body?: unknown, bearer?: string) => { const response = await (await relay()).dispatchFetch(ORIGINS.relay + path, { @@ -263,8 +270,9 @@ async function fixture( }); return { status: response.status, json: (await response.json()) as Record }; }; + let url: URL; try { - await worker.ready; + url = await worker.ready; } catch (error) { await worker.dispose(); await provider.close(); @@ -398,6 +406,9 @@ async function fixture( return { ...context, worker, + /** A relay socket's upgrade, from a Burrow: no Origin. */ + burrowSocket: (burrowToken: string) => + rawUpgrade(url, `${ORIGINS.relay}${WS_ROUTES.burrow}?${WS_TOKEN_PARAM}=${burrowToken}`, {}), provider, elevenLabs, browser, @@ -425,7 +436,6 @@ async function fixture( close: async () => { await worker.dispose(); await (await voiceWorker)?.dispose(); - await (await relayWorker)?.dispose(); await provider.close(); await context.close(); }, @@ -561,11 +571,12 @@ test("same-site requests fail, the sibling Workers' included; production exclude "/api/auth/revoke-sessions", ]) expect((await browser.request(path)).status).toBe(404); + // Straight to the account Worker: the relay beside it answers its own host. + const account = (await f.worker.getWorker(wrangler.account.name)) as unknown as { + fetch(url: string): Promise; + }; for (const sameSite of SAME_SITE) - expect( - (await f.worker.dispatchFetch(sameSite + "/api/auth/csrf")).status, - sameSite, - ).toBe(421); + expect((await account.fetch(sameSite + "/api/auth/csrf")).status, sameSite).toBe(421); await browser.email("real-clock@example.test"); expect( Math.abs( @@ -980,6 +991,8 @@ test("enrollment: only a recent admin login from this origin approves, and the B expect(listed.burrows.map(({ burrowId }) => burrowId)).toEqual([burrowId]); expect(Math.abs(Date.parse(listed.burrows[0].enrolledAt) - Date.now())).toBeLessThan(60_000); expect((await f.burrowCall(API_ROUTES.burrowSetupToken, undefined, burrowToken)).status).toBe(200); + const socket = await f.burrowSocket(burrowToken); + expect(socket.status).toBe(101); // Remove: this origin only; another account's Burrow is not found. expect((await admin.remove(burrowId, ORIGINS.relay)).status).toBe(403); @@ -990,7 +1003,11 @@ test("enrollment: only a recent admin login from this origin approves, and the B [foreign, (await other.session())!.user.id], ); expect((await admin.remove(foreign)).status).toBe(404); + expect(socket.socket!.closedWith()).toBeUndefined(); expect((await admin.remove(burrowId)).status).toBe(204); + // Its live relay socket closes as revoked, pushed from the account Worker. + expect((await socket.socket!.closed).code).toBe(WS_CLOSE_BURROW_REVOKED); + expect((await f.burrowSocket(burrowToken)).status).toBe(401); // A removed Burrow is gone, so its token opens nothing. expect(await f.burrowCall(API_ROUTES.burrowSetupToken, undefined, burrowToken)).toEqual({ status: 401, diff --git a/hosted/server/workers-runtime.d.ts b/hosted/server/workers-runtime.d.ts index c372bff10..1740e5b0c 100644 --- a/hosted/server/workers-runtime.d.ts +++ b/hosted/server/workers-runtime.d.ts @@ -27,7 +27,17 @@ interface DurableObjectState { getWebSockets(tag?: string): WorkerWebSocket[]; getTags(ws: WorkerWebSocket): string[]; setWebSocketAutoResponse(pair?: WebSocketRequestResponsePair): void; + /** When the runtime last answered `ws` through the auto-response pair, or null if never. */ + getWebSocketAutoResponseTimestamp(ws: WorkerWebSocket): Date | null; + /** The Worker's own named entrypoints, each callable as a loopback binding. */ + readonly exports: Record; + /** Run `callback` with no other event delivered to the object until it settles. */ + blockConcurrencyWhile(callback: () => Promise): Promise; readonly storage: { + get(key: string): Promise; + put(key: string, value: unknown): Promise; + list(): Promise>; + getAlarm(): Promise; setAlarm(scheduledTime: number): Promise; deleteAlarm(): Promise; deleteAll(): Promise; @@ -38,9 +48,27 @@ interface DurableObjectId { readonly name?: string; } -interface DurableObjectNamespace { +/** A Durable Object's stub: `fetch`, plus the RPC methods of the class `Stub` names. */ +type DurableObjectStub = { + fetch(request: Request): Promise; +} & Stub; + +interface DurableObjectNamespace { idFromName(name: string): DurableObjectId; - get(id: DurableObjectId): { fetch(request: Request): Promise }; + get(id: DurableObjectId): DurableObjectStub; +} + +/** The base class whose public methods a stub calls as RPC. */ +declare module "cloudflare:workers" { + export abstract class DurableObject { + protected readonly ctx: DurableObjectState; + protected readonly env: Env; + constructor(ctx: DurableObjectState, env: Env); + } + /** A named entrypoint, its public methods called as RPC. */ + export abstract class WorkerEntrypoint { + protected readonly env: Env; + } } /** A `ratelimits` binding. */ diff --git a/hosted/wrangler.jsonc b/hosted/wrangler.jsonc index 7e0d2c2d1..2538ab65a 100644 --- a/hosted/wrangler.jsonc +++ b/hosted/wrangler.jsonc @@ -31,6 +31,17 @@ "id": "5bfbeba081b4456f9c3cf36edbb20c1d" } ], + // The relay Worker's, so removing a Burrow closes its live socket. It is + // implemented there; this Worker creates no class of it. + "durable_objects": { + "bindings": [ + { + "name": "RELAY_ROOM", + "class_name": "RelayRoom", + "script_name": "dormouse-relay" + } + ] + }, "migrations": [ { "tag": "v1", diff --git a/hosted/wrangler.relay.jsonc b/hosted/wrangler.relay.jsonc index 60ef08f4b..65113bb35 100644 --- a/hosted/wrangler.relay.jsonc +++ b/hosted/wrangler.relay.jsonc @@ -35,15 +35,26 @@ { "name": "ONE_TIME_ROOM", "class_name": "OneTimeRoom" + }, + { + "name": "RELAY_ROOM", + "class_name": "RelayRoom" } ] }, + // Append-only: a deployed tag is never edited. "migrations": [ { "tag": "v1", "new_sqlite_classes": [ "OneTimeRoom" ] + }, + { + "tag": "v2", + "new_sqlite_classes": [ + "RelayRoom" + ] } ], "ratelimits": [ diff --git a/lib/src/remote/burrow/burrow-bounds.test.ts b/lib/src/remote/burrow/burrow-bounds.test.ts index ba5b86256..3c7e38206 100644 --- a/lib/src/remote/burrow/burrow-bounds.test.ts +++ b/lib/src/remote/burrow/burrow-bounds.test.ts @@ -24,6 +24,9 @@ import { MAX_E2E_CIPHERTEXT_LENGTH, MAX_CLIENT_ID_LENGTH, MAX_RELAY_TO_BURROW_FRAME_LENGTH, + RELAY_PING, + RELAY_PING_INTERVAL_MS, + RELAY_PONG, NoiseTransportSession, fromBase64Url, utf8Encode, @@ -103,6 +106,9 @@ function countCrypto() { }; } +/** The one timer an open relay socket always holds: its heartbeat. */ +const HEARTBEAT = 1; + describe('BurrowRuntime bounds', () => { let enrollment: BurrowEnrollment; let socket: FakeSocket; @@ -747,8 +753,9 @@ describe('BurrowRuntime bounds', () => { }); expect(burrow.trackedClientCount).toBe(0); expect(burrow.outstandingInvitationCount).toBe(0); - // Every deadline it held is gone, so nothing is left to wake the process. - expect(clock.armed).toBe(0); + // Every deadline it held is gone: what is left to wake the process is + // the relay socket's heartbeat alone. + expect(clock.armed).toBe(HEARTBEAT); // The abandoned challenge went with the record that named it, rather than // waiting on the issuer's own sweep. expect(burrow.pendingChallengeCount).toBe(0); @@ -779,7 +786,7 @@ describe('BurrowRuntime bounds', () => { expect(burrow.trackedClientCount).toBe(0); expect(burrow.pendingChallengeCount).toBe(0); expect(sessions.every((s) => s.disposed)).toBe(true); - expect(clock.armed).toBe(0); + expect(clock.armed).toBe(HEARTBEAT); }); it('expires a deadline that falls exactly on now', async () => { @@ -791,7 +798,7 @@ describe('BurrowRuntime bounds', () => { // reaper that arms for an instant it will not reap spins on that instant. clock.advance(1); expect(burrow.outstandingInvitationCount).toBe(0); - expect(clock.armed).toBe(0); + expect(clock.armed).toBe(HEARTBEAT); }); it('reaps sixteen silent sessions on the idle timeout, without a restart', async () => { @@ -950,6 +957,29 @@ describe('BurrowRuntime bounds', () => { expect(burrow.establishedSessionCount).toBe(0); }); + it('pings its relay socket every interval, holding a Relay that never answers to nothing', () => { + clock.advance(5 * RELAY_PING_INTERVAL_MS); + expect(socket.texts).toEqual(Array(5).fill(RELAY_PING)); + // An older Relay never pongs: the socket is as open as it was. + expect(burrow.status).toBe('connected'); + }); + + it('once a pong has arrived, a ping unanswered by the next one ends the socket', () => { + clock.advance(RELAY_PING_INTERVAL_MS); + socket.receiveRaw(RELAY_PONG); + // Answered on time, twice: still connected, and the pong reached no frame handler. + clock.advance(RELAY_PING_INTERVAL_MS); + socket.receiveRaw(RELAY_PONG); + clock.advance(RELAY_PING_INTERVAL_MS); + expect(burrow.status).toBe('connected'); + // Not answered before the next one is due: closed, through the ordinary + // close policy (this Burrow does not reconnect). + clock.advance(RELAY_PING_INTERVAL_MS); + expect(socket.readyState).toBe(3); + expect(burrow.status).toBe('stopped'); + expect(socket.texts).toHaveLength(3); + }); + it('leaves no timer armed once the Burrow stops', async () => { await burrow.mintInvitation(randomBase64Url(32), clock.now() + DEFAULT_PAIRING_TTL_MS); await establish('c1'); diff --git a/lib/src/remote/burrow/burrow-runtime.ts b/lib/src/remote/burrow/burrow-runtime.ts index e1e4e95ee..788363ccd 100644 --- a/lib/src/remote/burrow/burrow-runtime.ts +++ b/lib/src/remote/burrow/burrow-runtime.ts @@ -59,7 +59,13 @@ import { import type { BurrowEnrollment } from './enrollment'; import { createSerialQueue } from '../../host/remote/serial-queue'; import type { DirectPeering } from '../direct/direct-peer'; -import { closeCode, realTimer, type RemoteTimer, type RemoteWebSocket } from '../ws'; +import { + RelayHeartbeat, + closeCode, + realTimer, + type RemoteTimer, + type RemoteWebSocket, +} from '../ws'; import { loadBurrowAcl } from './acl'; import { EstablishedE2eSession, @@ -394,6 +400,8 @@ export class BurrowRuntime { #backoffMs = INITIAL_BACKOFF_MS; /** Cancels the armed reconnect, or null when none is armed. */ #cancelReconnect: (() => void) | null = null; + /** The open socket's heartbeat, or null while none is open. */ + #heartbeat: RelayHeartbeat | null = null; constructor(options: BurrowOptions) { this.#enrollment = options.enrollment; @@ -740,6 +748,7 @@ export class BurrowRuntime { this.#stopped = true; this.#status = 'stopped'; this.#clearReconnectTimer(); + this.#stopHeartbeat(); this.#dropTransientState(); // A stopped Burrow leaves no timer behind to wake the process it runs in. this.#clearReaper(); @@ -772,11 +781,14 @@ export class BurrowRuntime { if (this.#ws !== ws) return; this.#status = 'connected'; this.#backoffMs = INITIAL_BACKOFF_MS; + this.#heartbeat = new RelayHeartbeat(ws, this.#setTimer, () => this.#abandonSocket()); this.#reap(); }); ws.addEventListener('message', (ev) => { if (this.#ws !== ws) return; - this.#onFrame((ev as { data?: unknown }).data); + const data = (ev as { data?: unknown }).data; + if (this.#heartbeat?.read(data)) return; + this.#onFrame(data); }); ws.addEventListener('error', () => { // A `close` always follows; reconnection is handled there. @@ -793,6 +805,7 @@ export class BurrowRuntime { } #onClose(code: number | undefined): void { + this.#stopHeartbeat(); this.#dropTransientState(); if (this.#stopped) { this.#status = 'stopped'; @@ -825,6 +838,27 @@ export class BurrowRuntime { this.#cancelReconnect = null; } + #stopHeartbeat(): void { + this.#heartbeat?.stop(); + this.#heartbeat = null; + } + + /** + * End the socket here and now, through the ordinary close policy: its + * handlers are detached by the generation guard first, so nothing it still + * delivers is read. + */ + #abandonSocket(): void { + const ws = this.#ws; + this.#ws = null; + this.#onClose(undefined); + try { + ws?.close(); + } catch { + // Already closing. + } + } + /** * Connection-scoped state resets on a dropped socket; the ACL persists, and * invitations go with the socket (`docs/specs/remote-security-model.md` → @@ -933,14 +967,7 @@ export class BurrowRuntime { this.#frames.length >= MAX_QUEUED_RELAY_FRAMES || this.#frameChars + chars > MAX_QUEUED_RELAY_FRAME_CHARS ) { - const ws = this.#ws; - this.#ws = null; - this.#onClose(undefined); - try { - ws?.close(); - } catch { - // Already closing; its handlers are detached by the generation guard. - } + this.#abandonSocket(); return; } this.#frames.push({ frame, chars }); diff --git a/lib/src/remote/burrow/one-time-runtime.test.ts b/lib/src/remote/burrow/one-time-runtime.test.ts index 6b9659f18..bc9897d0f 100644 --- a/lib/src/remote/burrow/one-time-runtime.test.ts +++ b/lib/src/remote/burrow/one-time-runtime.test.ts @@ -19,9 +19,9 @@ import { MAX_ONE_TIME_FRAME_LENGTH, ONE_TIME_DIRECT_DEADLINE_MS, ONE_TIME_LINK_TTL_MS, - ONE_TIME_PING, - ONE_TIME_PING_INTERVAL_MS, - ONE_TIME_PONG, + RELAY_PING, + RELAY_PING_INTERVAL_MS, + RELAY_PONG, ONE_TIME_UNKNOWN_DEVICE_LABEL, TokenBucket, fromBase64Url, @@ -277,12 +277,12 @@ describe('OneTimeRuntime: the link', () => { it('pings the rendezvous while it waits, and neither counts nor parses the answers', async () => { makeRuntime(); const link = await openLink(); - clock.advance(ONE_TIME_PING_INTERVAL_MS); + clock.advance(RELAY_PING_INTERVAL_MS); const room = rendezvous.room(); - expect(room.burrow.sent).toEqual([ONE_TIME_PING]); - expect(room.burrow.received).toContain(ONE_TIME_PONG); + expect(room.burrow.sent).toEqual([RELAY_PING]); + expect(room.burrow.received).toContain(RELAY_PONG); // More answers than the message cap: none of them counts against it. - for (let i = 0; i <= MAX_ONE_TIME_FORWARDED; i += 1) fromRoom(ONE_TIME_PONG); + for (let i = 0; i <= MAX_ONE_TIME_FORWARDED; i += 1) fromRoom(RELAY_PONG); expect(runtime.state.status).toBe('waiting'); await joinPhone(link); }); diff --git a/lib/src/remote/burrow/one-time-runtime.ts b/lib/src/remote/burrow/one-time-runtime.ts index 386e85b56..54386ec11 100644 --- a/lib/src/remote/burrow/one-time-runtime.ts +++ b/lib/src/remote/burrow/one-time-runtime.ts @@ -22,7 +22,7 @@ import { NoiseTransportSession, ONE_TIME_DIRECT_DEADLINE_MS, ONE_TIME_LINK_TTL_MS, - ONE_TIME_PONG, + RELAY_PONG, ONE_TIME_WS_ROUTES, TokenBucket, WS_CLOSE_ONE_TIME_DEADLINE, @@ -376,7 +376,7 @@ export class OneTimeRuntime { #onMessage(raw: unknown): void { // A whole string, never JSON; the room answers pings itself and never // forwards or counts the answer, so neither does this. - if (raw === ONE_TIME_PONG) return; + if (raw === RELAY_PONG) return; if (!this.#link) { this.#onRoomMessage(raw); return; diff --git a/lib/src/remote/client/one-time-client.test.ts b/lib/src/remote/client/one-time-client.test.ts index bf654632d..0a92bd1b8 100644 --- a/lib/src/remote/client/one-time-client.test.ts +++ b/lib/src/remote/client/one-time-client.test.ts @@ -16,9 +16,9 @@ import { ONE_TIME_DENIAL_CODES, ONE_TIME_DIRECT_DEADLINE_MS, ONE_TIME_LINK_TTL_MS, - ONE_TIME_PING, - ONE_TIME_PING_INTERVAL_MS, - ONE_TIME_PONG, + RELAY_PING, + RELAY_PING_INTERVAL_MS, + RELAY_PONG, WS_CLOSE_ONE_TIME_DEADLINE, WS_CLOSE_ONE_TIME_EXPIRED, WS_CLOSE_ONE_TIME_PEER_GONE, @@ -276,9 +276,9 @@ describe('OneTimeClient: the rendezvous socket', () => { await flushUntil(() => (sockets.length > 0 && fromPhone().length > 0 ? true : undefined)); const socket = phoneSocket(); const parse = vi.spyOn(JSON, 'parse'); - socket.deliver(ONE_TIME_PONG); + socket.deliver(RELAY_PONG); // A whole-string compare, never a parse. - expect(parse.mock.calls.some(([text]) => text === ONE_TIME_PONG)).toBe(false); + expect(parse.mock.calls.some(([text]) => text === RELAY_PONG)).toBe(false); parse.mockRestore(); socket.deliver(new Uint8Array(8).buffer); socket.deliver('{"t":"one-time"'); @@ -299,16 +299,16 @@ describe('OneTimeClient: the rendezvous socket', () => { const burrow = await ScriptedBurrow.create(); const result = client.connectOnce(burrow.link, LABEL, () => {}); await burrow.answerInit(); - const pings = () => phoneSocket().sent.filter((data) => data === ONE_TIME_PING).length; - clock.advance(ONE_TIME_PING_INTERVAL_MS - 1); + const pings = () => phoneSocket().sent.filter((data) => data === RELAY_PING).length; + clock.advance(RELAY_PING_INTERVAL_MS - 1); expect(pings()).toBe(0); clock.advance(1); expect(pings()).toBe(1); - clock.advance(ONE_TIME_PING_INTERVAL_MS); + clock.advance(RELAY_PING_INTERVAL_MS); expect(pings()).toBe(2); phoneSocket().closeWith(WS_CLOSE_ONE_TIME_PEER_GONE); expect(await result).toEqual({ ok: false, message: ONE_TIME_ENDED_MESSAGE }); - clock.advance(ONE_TIME_PING_INTERVAL_MS); + clock.advance(RELAY_PING_INTERVAL_MS); expect(pings()).toBe(2); expect(clock.armed).toBe(0); }); diff --git a/lib/src/remote/client/one-time-client.ts b/lib/src/remote/client/one-time-client.ts index eb1b7ac58..58cdab9ae 100644 --- a/lib/src/remote/client/one-time-client.ts +++ b/lib/src/remote/client/one-time-client.ts @@ -19,7 +19,7 @@ import { NoiseTransportSession, ONE_TIME_DIRECT_DEADLINE_MS, ONE_TIME_EXPIRY_GRACE_MS, - ONE_TIME_PONG, + RELAY_PONG, ONE_TIME_ROOM_PARAM, ONE_TIME_WS_ROUTES, WS_CLOSE_ONE_TIME_DEADLINE, @@ -449,7 +449,7 @@ export class OneTimeClient implements RemoteAdapterClient { #onMessage(raw: unknown): void { // A whole string, never JSON: the room answers a ping itself and never // forwards the answer. - if (raw === ONE_TIME_PONG) return; + if (raw === RELAY_PONG) return; const frame = parseOneTimeFrame(raw); // The shared guard bounds every value before any is used as a key or // decoded; this phone runs it rather than trusting the room to have. diff --git a/lib/src/remote/client/pocket-client.test.ts b/lib/src/remote/client/pocket-client.test.ts index 83d64ba43..59190f48b 100644 --- a/lib/src/remote/client/pocket-client.test.ts +++ b/lib/src/remote/client/pocket-client.test.ts @@ -17,6 +17,9 @@ import { E2E_KEEPALIVE_INTERVAL_MS, ESTABLISHED_E2E_IDLE_TIMEOUT_MS, KEEPALIVE_BODY_SIZE, + RELAY_PING, + RELAY_PING_INTERVAL_MS, + RELAY_PONG, SETUP_TOKEN_INVALID_ERROR, formatPairingInvitationUrl, fromBase64Url, @@ -631,6 +634,10 @@ function fakeVisibility() { visible = next; for (const listener of listeners) listener(); }, + /** How many subscriptions are held. */ + get subscribers(): number { + return listeners.size; + }, }; } @@ -665,8 +672,11 @@ describe('keepalives on an established session', () => { // The kind byte, 32 zero bytes, and the Poly1305 tag: every keepalive is // this size, so the interval tells a timing observer nothing else. expect(fromBase64Url(sent[0]!.ct as string).length).toBe(1 + KEEPALIVE_BODY_SIZE + 16); - expect(timers.live).toHaveLength(1); - expect(timers.live[0]!.delayMs).toBe(E2E_KEEPALIVE_INTERVAL_MS); + // Re-armed, beside the relay socket's own heartbeat. + expect(timers.live.map(({ delayMs }) => delayMs)).toEqual([ + RELAY_PING_INTERVAL_MS, + E2E_KEEPALIVE_INTERVAL_MS, + ]); }); it('pauses while the page is hidden and sends one the moment it returns', async () => { @@ -679,10 +689,18 @@ describe('keepalives on an established session', () => { expect(timers.live).toHaveLength(0); expect(sentSince(harness, before)).toHaveLength(0); - // Back in front of the user: one immediately, then the interval again. + // Back in front of the user: one immediately, then the interval again, + // and the relay heartbeat with it. visibility.set(true); expect(sentSince(harness, before)).toHaveLength(1); - expect(timers.live).toHaveLength(1); + expect(timers.live).toHaveLength(2); + }); + + it('shares its one visibility subscription with the relay socket heartbeat', async () => { + const { harness, visibility } = await connected(); + expect(visibility.subscribers).toBe(1); + harness.client.close(); + expect(visibility.subscribers).toBe(0); }); it('stops when the session does', async () => { @@ -717,7 +735,8 @@ describe('keepalives on an established session', () => { // No keepalive into a session that no longer exists, and the app is told. expect(sentSince(harness, before)).toHaveLength(0); expect(gone).toHaveBeenCalledOnce(); - expect(timers.live).toHaveLength(0); + // Only the relay socket's heartbeat: the socket outlives the session. + expect(timers.live.map(({ delayMs }) => delayMs)).toEqual([RELAY_PING_INTERVAL_MS]); // And a request on the dead session fails rather than hanging. await expect(harness.client.write('s1', 'ls')).rejects.toThrow(); @@ -760,11 +779,77 @@ describe('keepalives on an established session', () => { expect(send).toHaveBeenCalledTimes(1); // And the next interval is still armed, so a socket that comes back is // keepalived again rather than silently reaped. - expect(timers.live).toHaveLength(1); + expect(timers.live).toHaveLength(2); send.mockRestore(); }); }); +describe('the relay socket heartbeat', () => { + /** A signed-in phone with its relay socket open, on a timer and visibility the test owns. */ + async function opened() { + const timers = fakeTimers(); + const visibility = fakeVisibility(); + const harness = await signedIn( + {}, + { setTimer: timers.setTimer, visibility: visibility.visibility }, + ); + const opening = harness.client.openSocket(); + harness.socket.open(); + await opening; + return { harness, timers, visibility }; + } + + it('pings every interval and holds a Relay that never answers to nothing', async () => { + const { harness, timers } = await opened(); + for (let i = 1; i <= 5; i += 1) { + timers.fireAt(RELAY_PING_INTERVAL_MS); + expect(harness.socket.texts).toHaveLength(i); + } + expect(harness.socket.texts.every((text) => text === RELAY_PING)).toBe(true); + // An older Relay never pongs: the socket is as open as it was. + expect(harness.client.socketOpen).toBe(true); + // A pong is the heartbeat's, never a frame the Client reads. + harness.socket.receiveRaw(RELAY_PONG); + expect(harness.client.socketOpen).toBe(true); + }); + + it('once a pong has arrived, a ping unanswered by the next one drops the socket', async () => { + const { harness, timers } = await opened(); + timers.fireAt(RELAY_PING_INTERVAL_MS); + harness.socket.receiveRaw(RELAY_PONG); + // Answered on time: still open. + timers.fireAt(RELAY_PING_INTERVAL_MS); + harness.socket.receiveRaw(RELAY_PONG); + timers.fireAt(RELAY_PING_INTERVAL_MS); + expect(harness.client.socketOpen).toBe(true); + // Not answered before the next one is due. + timers.fireAt(RELAY_PING_INTERVAL_MS); + expect(harness.client.socketOpen).toBe(false); + expect(harness.socket.readyState).toBe(3); + expect(timers.live).toHaveLength(0); + }); + + it('holds one visibility subscription while the socket is open, and none after', async () => { + const { harness, visibility } = await opened(); + expect(visibility.subscribers).toBe(1); + harness.client.close(); + expect(visibility.subscribers).toBe(0); + }); + + it('pauses while the page is hidden, and forgives the ping it was waiting on', async () => { + const { harness, timers, visibility } = await opened(); + timers.fireAt(RELAY_PING_INTERVAL_MS); + harness.socket.receiveRaw(RELAY_PONG); + timers.fireAt(RELAY_PING_INTERVAL_MS); + // Hidden with a ping outstanding: a throttled page cannot judge it. + visibility.set(false); + expect(timers.live).toHaveLength(0); + visibility.set(true); + timers.fireAt(RELAY_PING_INTERVAL_MS); + expect(harness.client.socketOpen).toBe(true); + }); +}); + // --- The direct path -------------------------------------------------------- describe('the direct path, end to end', () => { @@ -870,7 +955,7 @@ describe('the direct path, end to end', () => { const clientBefore = run.clientFrames().length; const sentBefore = run.network.offererChannel!.sent.length; - run.timers.fireAt(E2E_KEEPALIVE_INTERVAL_MS); + run.timers.fireLatestAt(E2E_KEEPALIVE_INTERVAL_MS); expect(run.clientFrames()).toHaveLength(clientBefore); const sent = run.network.offererChannel!.sent.slice(sentBefore); diff --git a/lib/src/remote/client/pocket-client.ts b/lib/src/remote/client/pocket-client.ts index e6265f6ad..9bd137c11 100644 --- a/lib/src/remote/client/pocket-client.ts +++ b/lib/src/remote/client/pocket-client.ts @@ -80,7 +80,7 @@ import { type PendingDeletionStore, } from './pocket-db'; import { SCAN_LABEL } from '../setup-copy'; -import type { RemoteWebSocket } from '../ws'; +import { RelayHeartbeat, realTimer, type RemoteTimer, type RemoteWebSocket } from '../ws'; import { ClientSessionCore, type ClientSessionCoreDeps } from './session-core'; import type { TerminalHandlers } from './remote-adapter'; @@ -117,7 +117,9 @@ export interface PocketStorage { /** * What a `PocketClient` is built from. The session core's own seams — - * `setTimer`, `visibility`, `createDirectPeer` — pass straight through to it. + * `setTimer`, `visibility`, `createDirectPeer` — pass straight through to it; + * the relay socket's heartbeat arms on the same `setTimer` and runs on the + * core's visibility. */ export interface PocketClientDeps extends Pick, 'setTimer' | 'visibility' | 'createDirectPeer'> { @@ -306,10 +308,13 @@ export class PocketClient { readonly #pendingDeletions: PendingDeletionStore; readonly #storage: PocketStorage; readonly #now: () => number; + readonly #setTimer: RemoteTimer; /** Everything a ceremony's frames and an established session do. */ readonly #core: ClientSessionCore; #ws: PocketSocket | null = null; + /** The open relay socket's heartbeat, or null while none is open. */ + #heartbeat: RelayHeartbeat | null = null; /** * The sign-in session: its token, and the account the Relay answered with * it — `'owner'` from a self-host Relay, the account's user id from Hosted. @@ -329,6 +334,7 @@ export class PocketClient { this.#pendingDeletions = deps.pendingDeletions; this.#storage = deps.storage ?? localStoragePocketStorage(); this.#now = deps.now ?? (() => Date.now()); + this.#setTimer = deps.setTimer ?? realTimer; this.#core = new ClientSessionCore({ sendFrame: (route, step, ciphertext) => this.#sendE2e(route, step, ciphertext), messages: { @@ -654,7 +660,9 @@ export class PocketClient { const isCurrent = () => this.#ws === ws; ws.addEventListener('message', (ev) => { if (!isCurrent()) return; - this.#onFrame((ev as { data?: unknown }).data); + const data = (ev as { data?: unknown }).data; + if (this.#heartbeat?.read(data)) return; + this.#onFrame(data); }); ws.addEventListener('close', () => this.#onClose(ws)); return new Promise((resolve, reject) => { @@ -663,6 +671,7 @@ export class PocketClient { reject(new Error('relay socket superseded')); return; } + this.#startHeartbeat(ws); resolve(); }); ws.addEventListener('error', () => reject(new Error('relay socket error'))); @@ -1073,6 +1082,30 @@ export class PocketClient { } } + /** + * Ping the relay socket while the page is visible, on the session core's + * visibility as keepalives run (`docs/specs/pocket-app.md`); a socket that + * stops answering is a drop, though no close arrived. + */ + #startHeartbeat(ws: PocketSocket): void { + this.#stopHeartbeat(); + this.#heartbeat = new RelayHeartbeat(ws, this.#setTimer, () => { + this.#onClose(ws); + try { + ws.close(); + } catch { + // already closing + } + }); + this.#core.setSocketHeartbeat(this.#heartbeat); + } + + #stopHeartbeat(): void { + this.#heartbeat?.stop(); + this.#heartbeat = null; + this.#core.setSocketHeartbeat(null); + } + #onClose(ws: PocketSocket): void { // Generation guard, and the whole test for "was this close intentional?": // `close()` tears down and nulls #ws *before* calling `ws.close()`, and a @@ -1093,6 +1126,7 @@ export class PocketClient { */ #teardown(reason: string, { notifyGone }: { notifyGone: boolean }): void { this.#ws = null; // never reuse a closed socket; openSocket() makes a fresh one + this.#stopHeartbeat(); this.#core.endSession(reason, { notifyGone }); } diff --git a/lib/src/remote/client/session-core.ts b/lib/src/remote/client/session-core.ts index 4d53b9935..d3074f717 100644 --- a/lib/src/remote/client/session-core.ts +++ b/lib/src/remote/client/session-core.ts @@ -40,7 +40,7 @@ import { } from 'remote-lib-common'; import { DirectEndpoint } from '../direct/direct-endpoint'; import type { DirectPeerFactory } from '../direct/direct-peer'; -import { realTimer, type RemoteTimer } from '../ws'; +import { realTimer, type RelayHeartbeat, type RemoteTimer } from '../ws'; import type { RemoteAdapterClient, TerminalHandlers } from './remote-adapter'; /** @@ -144,8 +144,11 @@ export class ClientSessionCore implements RemoteAdapter #onTransportChanged: | ((path: DirectPath, cause: DirectRelayCause | null) => void) | null = null; - /** Cancels the armed keepalive, and the visibility subscription behind it. */ + /** Cancels the armed keepalive. */ #cancelKeepalive: (() => void) | null = null; + /** The owner's relay-socket heartbeat, run beside the keepalives; see {@link setSocketHeartbeat}. */ + #heartbeat: RelayHeartbeat | null = null; + /** Cancels the one visibility subscription, held while a session or a heartbeat runs. */ #cancelVisibility: (() => void) | null = null; /** @@ -497,12 +500,40 @@ export class ClientSessionCore implements RemoteAdapter * ([pocket-app.md](../../../../docs/specs/pocket-app.md)). */ #startKeepalives(): void { - this.#stopKeepalives(); - this.#cancelVisibility = this.#visibility.subscribe(() => { - if (this.#established && this.#visibility.isVisible()) this.sendKeepalive(); - // Re-arms while visible and cancels while hidden; one place decides. - this.#armKeepalive(); - }); + this.#watchVisibility(); + this.#armKeepalive(); + } + + /** + * Run the owner's relay-socket heartbeat — or, with `null`, none — on this + * core's visibility, as keepalives run: stopped while the page is hidden, + * whose throttled timers could not judge a pong, and resumed with the + * outstanding ping forgiven when it returns. The owner starts and stops the + * heartbeat itself. + */ + setSocketHeartbeat(heartbeat: RelayHeartbeat | null): void { + this.#heartbeat = heartbeat; + if (heartbeat && !this.#visibility.isVisible()) heartbeat.stop(); + this.#watchVisibility(); + } + + /** The one visibility subscription: held while a session or a heartbeat runs, and only then. */ + #watchVisibility(): void { + const wanted = this.#established !== null || this.#heartbeat !== null; + if (wanted && !this.#cancelVisibility) { + this.#cancelVisibility = this.#visibility.subscribe(() => this.#onVisibilityChange()); + } else if (!wanted && this.#cancelVisibility) { + this.#cancelVisibility(); + this.#cancelVisibility = null; + } + } + + #onVisibilityChange(): void { + const visible = this.#visibility.isVisible(); + if (visible) this.#heartbeat?.resume(); + else this.#heartbeat?.stop(); + if (this.#established && visible) this.sendKeepalive(); + // Re-arms while visible and cancels while hidden; one place decides. this.#armKeepalive(); } @@ -521,11 +552,6 @@ export class ClientSessionCore implements RemoteAdapter this.#cancelKeepalive = null; } - #stopKeepalives(): void { - this.#cancelKeepaliveTimer(); - this.#cancelVisibility?.(); - this.#cancelVisibility = null; - } /** * One protocol-v1 message on the established session, chunked as it needs. @@ -659,7 +685,7 @@ export class ClientSessionCore implements RemoteAdapter /** Erase every session's cipher state; a new ceremony starts from a handshake. */ disposeSession(): void { - this.#stopKeepalives(); + this.#cancelKeepaliveTimer(); // The peer connection is this session's: every disposal path closes it, so // none can outlive the session that authorized it. const direct = this.#established?.direct ?? null; @@ -670,6 +696,7 @@ export class ClientSessionCore implements RemoteAdapter // stayed relayed would sit in the indicator through the whole of the next. if (direct?.relayCause) this.#onTransportChanged?.('relay', null); this.#established = null; + this.#watchVisibility(); } /** Fail every awaited ceremony frame and in-flight request (avoids hangs). */ diff --git a/lib/src/remote/one-time-rendezvous.ts b/lib/src/remote/one-time-rendezvous.ts index 96fc91863..73d8a9976 100644 --- a/lib/src/remote/one-time-rendezvous.ts +++ b/lib/src/remote/one-time-rendezvous.ts @@ -7,13 +7,9 @@ * since only the phone's may throw. */ -import { - MAX_ONE_TIME_FRAME_LENGTH, - ONE_TIME_PING, - ONE_TIME_PING_INTERVAL_MS, -} from 'remote-lib-common'; +import { MAX_ONE_TIME_FRAME_LENGTH } from 'remote-lib-common'; -import type { RemoteTimer, RemoteWebSocket } from './ws'; +import { RelayHeartbeat, type RemoteTimer, type RemoteWebSocket } from './ws'; /** The close an end ends its own rendezvous socket with. */ const NORMAL_CLOSURE = 1000; @@ -42,7 +38,7 @@ export class RendezvousHold { * close. */ #ws: RemoteWebSocket | null = null; - #cancelPing: (() => void) | null = null; + #heartbeat: RelayHeartbeat | null = null; constructor(setTimer: RemoteTimer) { this.#setTimer = setTimer; @@ -63,26 +59,21 @@ export class RendezvousHold { return this.#ws === ws; } - /** Keep the socket's path alive while it is open; the room answers without waking. */ + /** + * Keep the socket's path alive while it is open: the relay socket's + * heartbeat, which the room answers without waking, holding the room to no + * deadline — the end's own deadlines bound it. + */ armPing(ws: RemoteWebSocket): void { - this.#cancelPing = this.#setTimer(() => { - this.#cancelPing = null; - if (this.#ws !== ws) return; - try { - ws.send(ONE_TIME_PING); - } catch { - // socket mid-close - } - this.armPing(ws); - }, ONE_TIME_PING_INTERVAL_MS); + this.#heartbeat = new RelayHeartbeat(ws, this.#setTimer); } /** Stop reading the socket: nothing it says or does from here is an event. */ detach(): RemoteWebSocket | null { const ws = this.#ws; this.#ws = null; - this.#cancelPing?.(); - this.#cancelPing = null; + this.#heartbeat?.stop(); + this.#heartbeat = null; return ws; } diff --git a/lib/src/remote/test-fake-socket.ts b/lib/src/remote/test-fake-socket.ts index a4b7eae44..b6c9f05f5 100644 --- a/lib/src/remote/test-fake-socket.ts +++ b/lib/src/remote/test-fake-socket.ts @@ -39,6 +39,8 @@ export class FakeSocket implements RemoteWebSocket { */ closeEmits = true; readonly sent: Array> = []; + /** Every message sent that is not JSON: the relay heartbeat's pings. */ + readonly texts: string[] = []; /** * Called with every frame this socket is asked to send. The seam the relay * stub (`test-relay.ts`) bridges two of these sockets through; without it a @@ -53,7 +55,13 @@ export class FakeSocket implements RemoteWebSocket { } send(data: string): void { - const frame = JSON.parse(data) as Record; + let frame: Record; + try { + frame = JSON.parse(data) as Record; + } catch { + this.texts.push(data); + return; + } this.sent.push(frame); this.onSend?.(frame); } diff --git a/lib/src/remote/test-rendezvous.ts b/lib/src/remote/test-rendezvous.ts index d15320e02..dbda66e73 100644 --- a/lib/src/remote/test-rendezvous.ts +++ b/lib/src/remote/test-rendezvous.ts @@ -18,7 +18,7 @@ * - **Strings forwarded verbatim, never parsed**, each counted toward * `MAX_ONE_TIME_FORWARDED` across both directions; a message with nobody on * the other end yet is dropped uncounted. - * - **`ONE_TIME_PING` answered with `ONE_TIME_PONG`**, never forwarded or + * - **`RELAY_PING` answered with `RELAY_PONG`**, never forwarded or * counted. * - **A non-string, an oversize string, or one past the cap closes both * `4015`.** @@ -43,8 +43,8 @@ import { NoiseTransportSession, ONE_TIME_EXPIRY_GRACE_MS, ONE_TIME_LINK_TTL_MS, - ONE_TIME_PING, - ONE_TIME_PONG, + RELAY_PING, + RELAY_PONG, ONE_TIME_ROOM_PARAM, ONE_TIME_WS_ROUTES, WS_CLOSE_ONE_TIME_DEADLINE, @@ -246,8 +246,8 @@ export function createTestRendezvous(options: TestRendezvousOptions = {}): TestR const forward = (room: Room, from: RendezvousSocket, data: unknown): void => { if (room.deleted) return; - if (data === ONE_TIME_PING) { - from.deliver(ONE_TIME_PONG); + if (data === RELAY_PING) { + from.deliver(RELAY_PONG); return; } if (typeof data !== 'string' || data.length > MAX_ONE_TIME_FRAME_LENGTH) { diff --git a/lib/src/remote/test-timers.ts b/lib/src/remote/test-timers.ts index 54bcb2321..cfe214b82 100644 --- a/lib/src/remote/test-timers.ts +++ b/lib/src/remote/test-timers.ts @@ -18,6 +18,11 @@ export interface FakeTimers { fire(): void; /** Fire the one armed for `delayMs`, where more than one deadline is live. */ fireAt(delayMs: number): void; + /** + * Fire the one armed for `delayMs` most recently, where two share it: a + * session's keepalive and its relay socket's heartbeat run on one interval. + */ + fireLatestAt(delayMs: number): void; } export function fakeTimers(): FakeTimers { @@ -49,6 +54,12 @@ export function fakeTimers(): FakeTimers { if (index < 0) throw new Error(`no timer armed for ${delayMs}ms`); take(index)(); }, + fireLatestAt(delayMs: number): void { + let index = live.length - 1; + while (index >= 0 && live[index]!.delayMs !== delayMs) index -= 1; + if (index < 0) throw new Error(`no timer armed for ${delayMs}ms`); + take(index)(); + }, }; } diff --git a/lib/src/remote/ws.test.ts b/lib/src/remote/ws.test.ts new file mode 100644 index 000000000..75d89e6b0 --- /dev/null +++ b/lib/src/remote/ws.test.ts @@ -0,0 +1,59 @@ +import { describe, expect, it, vi } from 'vitest'; +import { RELAY_PING, RELAY_PONG } from 'remote-lib-common'; + +import { fakeTimers } from './test-timers'; +import { RelayHeartbeat, type RemoteWebSocket } from './ws'; + +/** A socket that keeps what it was sent. */ +function socket() { + const sent: string[] = []; + const ws: RemoteWebSocket = { + send: (data) => void sent.push(data), + close: () => {}, + addEventListener: () => {}, + readyState: 1, + }; + return { ws, sent }; +} + +// The relay socket's heartbeat (docs/specs/relay.md -> "Routing"); the Burrow's +// and Pocket's own suites drive it through their sockets. +describe('RelayHeartbeat', () => { + it('reads the pong, and only the pong, as its own', () => { + const timers = fakeTimers(); + const heartbeat = new RelayHeartbeat(socket().ws, timers.setTimer, () => {}); + expect(heartbeat.read(RELAY_PONG)).toBe(true); + for (const data of [RELAY_PING, '"pong"', '{}', undefined, new ArrayBuffer(4)]) + expect(heartbeat.read(data)).toBe(false); + }); + + it('with onDead, ends a socket that stops answering once it has answered', () => { + const timers = fakeTimers(); + const { ws, sent } = socket(); + const onDead = vi.fn(); + const heartbeat = new RelayHeartbeat(ws, timers.setTimer, onDead); + timers.fire(); + timers.fire(); + // Never answered: a Relay that never pongs is held to nothing. + expect(onDead).not.toHaveBeenCalled(); + heartbeat.read(RELAY_PONG); + timers.fire(); + timers.fire(); + expect(onDead).toHaveBeenCalledOnce(); + expect(sent).toEqual([RELAY_PING, RELAY_PING, RELAY_PING]); + expect(timers.live).toHaveLength(0); + }); + + it('without onDead, only keeps the path alive', () => { + const timers = fakeTimers(); + const { ws, sent } = socket(); + const heartbeat = new RelayHeartbeat(ws, timers.setTimer); + timers.fire(); + heartbeat.read(RELAY_PONG); + for (let i = 0; i < 4; i += 1) timers.fire(); + expect(sent).toHaveLength(5); + expect(timers.live).toHaveLength(1); + heartbeat.stop(); + expect(timers.live).toHaveLength(0); + }); +}); diff --git a/lib/src/remote/ws.ts b/lib/src/remote/ws.ts index 1cdd0c377..044016c68 100644 --- a/lib/src/remote/ws.ts +++ b/lib/src/remote/ws.ts @@ -1,3 +1,5 @@ +import { RELAY_PING, RELAY_PING_INTERVAL_MS, RELAY_PONG } from 'remote-lib-common'; + /** * The minimal WebSocket surface the remote client and burrow actually use — just * enough to send, close, and listen, so tests can inject a fake in place of a @@ -33,3 +35,72 @@ export const realTimer: RemoteTimer = (run, delayMs) => { const timer = setTimeout(run, delayMs); return () => clearTimeout(timer); }; + +/** + * A relay socket's heartbeat (`docs/specs/relay.md` -> "Routing"): a + * {@link RELAY_PING} every {@link RELAY_PING_INTERVAL_MS} while it runs. With + * `onDead`, once a pong has arrived on the socket, a ping still unanswered when + * the next one is due ends the socket through it; until then nothing is + * enforced, so a Relay that never answers costs nothing. Without `onDead` it + * only keeps the path alive: a one-time rendezvous is bounded by its own + * deadlines. + */ +export class RelayHeartbeat { + readonly #ws: RemoteWebSocket; + readonly #setTimer: RemoteTimer; + readonly #onDead: (() => void) | null; + #cancel: (() => void) | null = null; + /** A pong has arrived on this socket: from now on each ping must be answered. */ + #enforced = false; + #answered = true; + + constructor(ws: RemoteWebSocket, setTimer: RemoteTimer, onDead?: () => void) { + this.#ws = ws; + this.#setTimer = setTimer; + this.#onDead = onDead ?? null; + this.#arm(); + } + + /** + * Whether `data`, a message off the socket, is the Relay's {@link RELAY_PONG}: + * the heartbeat's own, which the caller then never reads as a frame. + */ + read(data: unknown): boolean { + if (data !== RELAY_PONG) return false; + this.#enforced = true; + this.#answered = true; + return true; + } + + /** + * Ping again from now, the last ping forgiven: after {@link stop} while a + * page was hidden, whose throttled timers could not have read a pong. + */ + resume(): void { + this.stop(); + this.#answered = true; + this.#arm(); + } + + stop(): void { + this.#cancel?.(); + this.#cancel = null; + } + + #arm(): void { + this.#cancel = this.#setTimer(() => { + this.#cancel = null; + if (this.#onDead && this.#enforced && !this.#answered) { + this.#onDead(); + return; + } + this.#answered = false; + try { + this.#ws.send(RELAY_PING); + } catch { + // socket mid-close + } + this.#arm(); + }, RELAY_PING_INTERVAL_MS); + } +} diff --git a/relay/scripts/fake-burrow.mjs b/relay/scripts/fake-burrow.mjs index 626a65ae1..d49818ee9 100644 --- a/relay/scripts/fake-burrow.mjs +++ b/relay/scripts/fake-burrow.mjs @@ -19,7 +19,7 @@ import { API_ROUTES, formatPairingInvitationUrl, generateNoiseKeyPair } from 'remote-lib-common'; import { SetupPasswordStore } from '../dist/state.js'; -import { FakeBurrow } from '../test/harness/fake-burrow.mjs'; +import { FakeBurrow } from '../../remote-lib-common/test/harness/fake-burrow.mjs'; import { devStateDir } from './dev-paths.mjs'; const relayUrl = (process.argv[2] ?? 'http://localhost:3000').replace(/\/$/, ''); diff --git a/relay/src/app.ts b/relay/src/app.ts index 6d13710b7..0a8c150a2 100644 --- a/relay/src/app.ts +++ b/relay/src/app.ts @@ -16,11 +16,16 @@ import type { NodeWebSocket } from '@hono/node-ws'; import { serveStatic } from '@hono/node-server/serve-static'; import { API_ROUTES, + MAX_RELAY_FRAME_BYTES, + RELAY_IDLE_TIMEOUT_MS, + UNKNOWN_BURROW_TOKEN_ERROR, + WS_CLOSE_TRY_AGAIN_LATER, + WS_CLOSE_UNAUTHORIZED, + WS_CLOSE_UNAUTHORIZED_REASON, + WS_CLOSE_IDLE, + WS_CLOSE_IDLE_REASON, DELIVERY_ID_LENGTH, - E2E_ID_LENGTH, ChallengeIssuer, - MAX_CLIENT_ID_LENGTH, - MAX_E2E_CIPHERTEXT_LENGTH, MAX_PUSH_QUERY_DELIVERY_IDS, MAX_SEALED_PUSH_LENGTH, SELFHOST_ACCOUNT_ID, @@ -91,12 +96,7 @@ import type { } from 'remote-lib-common'; import { invalidateEnrollOffer, redeemEnrollToken } from './enroll-token.js'; -import { - RelayHub, - WS_CLOSE_TRY_AGAIN_LATER, - WS_CLOSE_UNAUTHORIZED, - WS_CLOSE_UNAUTHORIZED_REASON, -} from './relay.js'; +import { RelayHub } from './relay.js'; import type { ClientConn, BurrowConn } from './relay.js'; import { secretEquals } from './secrets.js'; import { SetupTokenIssuer } from './setup-token.js'; @@ -219,24 +219,6 @@ export const BURROW_REVOCATION_SWEEP_MS = 60_000; * Client sessions and pings the rest. */ export const RELAY_SWEEP_MS = 30_000; -/** - * How long a relay socket may go unheard-from before it is closed. Three sweeps - * of silence: a live peer answers the first ping, so reaching this means the - * connection is half-open, not idle. Generous against a phone whose radio has - * dozed, which reconnects anyway. - */ -export const RELAY_IDLE_TIMEOUT_MS = 3 * RELAY_SWEEP_MS; -/** A socket closed for silence, not for anything it did. */ -const WS_CLOSE_IDLE = 1001; -const WS_CLOSE_IDLE_REASON = 'no response to heartbeat'; -/** - * The largest frame `ws` may buffer for us. Derived from the wire bounds the - * relay's own guards enforce — a maximal `ct` plus the envelope around it — - * because without it `ws` buffers up to 100 MiB before any guard has run. - * `MAX_CLIENT_ID_LENGTH` is in here because a Burrow frame carries one. - */ -export const MAX_RELAY_FRAME_BYTES = - MAX_E2E_CIPHERTEXT_LENGTH + MAX_CLIENT_ID_LENGTH + 2 * E2E_ID_LENGTH + 1024; /** A small fixed delay on a rejected credential. */ const CREDENTIAL_FAILURE_DELAY_MS = 250; @@ -1200,7 +1182,7 @@ export function createApp(config: AppConfig): CreatedApp { async (c, next) => { const token = c.req.query(WS_TOKEN_PARAM); const burrow = token ? await burrowStore.findByToken(token) : undefined; - if (!burrow) return c.json({ error: 'unknown burrow token' }, 401); + if (!burrow) return c.json({ error: UNKNOWN_BURROW_TOKEN_ERROR }, 401); c.set('burrow', burrow); return next(); }, diff --git a/relay/src/relay.ts b/relay/src/relay.ts index 9016ea9fd..a63d7ca1f 100644 --- a/relay/src/relay.ts +++ b/relay/src/relay.ts @@ -3,23 +3,24 @@ * Burrow authority, replacement, and routing contracts. */ -import { randomBytes } from 'node:crypto'; - import { + MAX_RELAY_CLIENT_SOCKETS, + RELAY_PING, + RELAY_PONG, + WS_CLOSE_UNAUTHORIZED, + WS_CLOSE_UNAUTHORIZED_REASON, WS_CLOSE_BURROW_REPLACED, WS_CLOSE_BURROW_REPLACED_REASON, WS_CLOSE_BURROW_REVOKED, WS_CLOSE_BURROW_REVOKED_REASON, - isE2eClientFrame, - isE2eBurrowFrame, - toBase64Url, -} from 'remote-lib-common'; -import type { - ClientFrame, - BurrowFrame, - RelayToClientFrame, - RelayToBurrowFrame, + newClientId, + offlineError, + readBurrowFrame, + readClientFrame, + toBurrowEnvelope, + toClientEnvelope, } from 'remote-lib-common'; +import type { RelayToClientFrame, RelayToBurrowFrame } from 'remote-lib-common'; /** * The slice of a WebSocket the hub actually uses. `WSContext` from @@ -61,27 +62,6 @@ export interface ClientConn { burrowId: string | null; } -/** - * How many Client sockets this process will hold at once. - * - * `/ws/client` needs a session token, and one account's phones are a handful, - * so this is far above real use — but without it a token-holder opens sockets - * until the process runs out, and a half-open TCP connection keeps its entry - * until the OS gives up. The heartbeat in `app.ts` is the other half of that. - */ -export const MAX_RELAY_CLIENT_SOCKETS = 64; - -/** Refused because the process is already holding {@link MAX_RELAY_CLIENT_SOCKETS}. */ -export const WS_CLOSE_TRY_AGAIN_LATER = 1013; - -/** - * The session behind this socket is gone. The same pair the `/ws/client` - * upgrade answers with, so a socket closed by the sweep is indistinguishable - * from one refused at the door and Pocket needs no second recovery. - */ -export const WS_CLOSE_UNAUTHORIZED = 1008; -export const WS_CLOSE_UNAUTHORIZED_REASON = 'unauthorized'; - export class RelayHub { readonly #burrows = new Map(); readonly #clients = new Map(); @@ -146,10 +126,11 @@ export class RelayHub { // current would carry ciphertext from the dead burrow process into a binding // the replacement never made. if (this.#burrows.get(burrow.burrowId) !== burrow) return; - const frame = parseFrame(raw); + if (answeredPing(burrow.socket, raw)) return; // The shape guard bounds `clientId` before it is used as a map key, and the // ciphertext before it is copied onto another socket. - if (!frame || !isE2eBurrowFrame(frame)) return; + const frame = readBurrowFrame(raw); + if (!frame) return; // Every burrow frame addresses a specific client; if it has already gone, // there is nothing to route. const client = this.#clients.get(frame.clientId); @@ -161,14 +142,7 @@ export class RelayHub { // No `authorized` gate: the relay never learns whether the Burrow authorized // anything, so the binding checked above is the whole routing rule // (relay.md -> "Routing"). - this.#toClient(client, { - t: 'e2e', - burrowId: burrow.burrowId, - kind: frame.kind, - id: frame.id, - step: frame.step, - ct: frame.ct, - }); + this.#toClient(client, toClientEnvelope(burrow.burrowId, frame)); } /** @@ -214,7 +188,7 @@ export class RelayHub { */ registerClient(socket: RelaySocket, session: RelaySession): ClientConn | null { if (this.#clients.size >= MAX_RELAY_CLIENT_SOCKETS) return null; - const clientId = toBase64Url(randomBytes(16)); + const clientId = newClientId(); const conn: ClientConn = { clientId, socket, session, burrowId: null }; this.#clients.set(clientId, conn); return conn; @@ -250,22 +224,16 @@ export class RelayHub { // the session the sweep just expired, which is the whole point of expiring // it (`docs/specs/relay.md` -> "Routing"). if (this.#clients.get(client.clientId) !== client) return; - const frame = parseFrame(raw); - if (!frame || typeof frame.t !== 'string') { - this.#toClient(client, { t: 'error', error: 'malformed frame' }); - return; - } - if (frame.t !== 'e2e') { - this.#toClient(client, { t: 'error', error: 'unknown frame type' }); - return; - } + if (answeredPing(client.socket, raw)) return; // The envelope the end-to-end protocol rides in: an `init` binds, and // everything after it is forwarded within that binding (relay.md -> // Relay). Never decoded here. - if (!isE2eClientFrame(frame)) { - this.#toClient(client, { t: 'error', error: 'malformed e2e frame' }); + const read = readClientFrame(raw); + if ('error' in read) { + this.#toClient(client, read.error); return; } + const { frame } = read; const burrow = this.#resolveBurrow(client, frame.burrowId); if (!burrow) return; if (frame.step === 'init') { @@ -275,15 +243,7 @@ export class RelayHub { // not bound to, so there is nothing to forward it to. return; } - this.#toBurrow(burrow, { - t: 'e2e', - clientId: client.clientId, - burrowId: frame.burrowId, - kind: frame.kind, - id: frame.id, - step: frame.step, - ct: frame.ct, - }); + this.#toBurrow(burrow, toBurrowEnvelope(client.clientId, frame)); } /** @@ -329,7 +289,7 @@ export class RelayHub { #resolveBurrow(client: ClientConn, burrowId: string): BurrowConn | null { const burrow = this.#burrows.get(burrowId); if (!burrow) { - this.#toClient(client, { t: 'error', error: `burrow ${burrowId} is offline` }); + this.#toClient(client, offlineError(burrowId)); return null; } return burrow; @@ -349,15 +309,19 @@ export class RelayHub { // --------------------------------------------------------------------------- // Helpers -/** Parse a raw WS text frame; `null` if it is not a JSON object. */ -function parseFrame(raw: string): (T & { t?: unknown }) | null { +/** + * Answer {@link RELAY_PING} with {@link RELAY_PONG}, as the Hosted Relay's + * auto-response does: the ping is the whole message, never parsed or + * forwarded. True when `raw` was the ping. + */ +function answeredPing(socket: RelaySocket, raw: string): boolean { + if (raw !== RELAY_PING) return false; try { - const parsed: unknown = JSON.parse(raw); - if (typeof parsed !== 'object' || parsed === null) return null; - return parsed as T & { t?: unknown }; + socket.send(RELAY_PONG); } catch { - return null; + // mid-close } + return true; } /** Serialize and send, swallowing errors from a socket that is mid-close. */ diff --git a/relay/test/e2e-ceremony.test.mjs b/relay/test/e2e-ceremony.test.mjs index 3b488bf77..899a8b2d6 100644 --- a/relay/test/e2e-ceremony.test.mjs +++ b/relay/test/e2e-ceremony.test.mjs @@ -34,8 +34,8 @@ import { signin, startRelay, } from './helpers.mjs'; -import { FakeClient } from './harness/fake-client.mjs'; -import { FakeBurrow } from './harness/fake-burrow.mjs'; +import { FakeClient } from '../../remote-lib-common/test/harness/fake-client.mjs'; +import { FakeBurrow } from '../../remote-lib-common/test/harness/fake-burrow.mjs'; import { randomSecret } from '../../remote-lib-common/test/harness/actors.mjs'; const BURROW_LABEL = 'Ned Laptop'; diff --git a/relay/test/e2e-relay.test.mjs b/relay/test/e2e-relay.test.mjs index 015b821f6..fbdf8f696 100644 --- a/relay/test/e2e-relay.test.mjs +++ b/relay/test/e2e-relay.test.mjs @@ -6,8 +6,10 @@ * What it proves, in the order the scope asks for it * (docs/specs/remote-security-model.md -> `## Future` -> **Scope: * e2e-client-burrow**, stage 3): prologue and transcript binding, directional - * cipher states, counters, framing, teardown, relay opacity, tamper rejection, - * and the relay's own bounds. The framing in isolation is + * cipher states, counters, framing, and tamper rejection. Its routing half — + * teardown, relay opacity, the binding, and the relay's own bounds — is the + * cases every Relay passes (`remote-lib-common/test/harness/relay-parity.mjs`), + * registered at the end. The framing in isolation is * `remote-lib-common/test/noise-transport.test.mjs`. */ @@ -16,66 +18,18 @@ import assert from 'node:assert/strict'; import { MAX_E2E_CIPHERTEXT_LENGTH, - WS_CLOSE_BURROW_REPLACED, e2eConnectionPrologue, fromBase64Url, generateNoiseKeyPair, toBase64Url, - utf8Encode, } from 'remote-lib-common'; import { until } from './helpers.mjs'; import { e2eFixture, establish, flip, newE2eId, watch } from './harness/e2e.mjs'; +import { e2eCases } from '../../remote-lib-common/test/harness/relay-parity.mjs'; const EMPTY = new Uint8Array(0); -/** Every frame the relay handled: what both peers sent and what it delivered. */ -function relayView(...peers) { - return JSON.stringify(peers.flatMap((peer) => [...peer.sent, ...peer.frames])); -} - -test('an established session round-trips every transport kind through the relay', async () => { - const fixture = await e2eFixture(); - const { burrow, client, clientStatic } = fixture; - const opens = []; - burrow.on('e2e-open', (ev) => opens.push(ev)); - try { - await establish(fixture); - // The connection handshake: both sides agree on the transcript, and IK - // authenticated the Client's static — the key the ACL conjunction matched. - const entry = opens.at(-1); - assert.deepEqual(entry.session.handshakeHash, client.session.handshakeHash); - assert.equal(entry.clientStaticPublicKey, toBase64Url(clientStatic.publicKey)); - - // Client → Burrow, all three kinds. - const seen = watch(burrow); - const payload = utf8Encode('terminal.write rides in here'); - client.sendKeepalive(); - client.sendControl({ presence: 'proof' }); - client.sendApp(payload); - await until(() => seen.receipts.length === 3); - assert.equal(seen.receipts[0].receipt.kind, 'keepalive'); - assert.deepEqual(seen.receipts[1].receipt, { kind: 'control', value: { presence: 'proof' } }); - assert.deepEqual(seen.receipts[2].receipt.messages, [payload]); - - // Burrow → Client, on the other direction's cipher state. - const reply = utf8Encode('terminal.data rides back'); - burrow.e2eSendApp(entry.clientId, reply); - const frame = await client.nextTransport(); - assert.equal(frame.burrowId, fixture.enrollment.burrowId, 'the relay stamps burrowId'); - assert.deepEqual(client.receiveFrame(frame).messages, [reply]); - - // The envelope is the whole surface: an established session opens no other - // pipe, and every other frame type is simply unknown. - const burrowFramesBefore = burrow.frames.length; - client.sendFrame({ t: 'msg', data: { forbidden: true } }); - const refusal = await client.waitFor((f) => f.t === 'error'); - assert.equal(refusal.error, 'unknown frame type'); - assert.equal(burrow.frames.length, burrowFramesBefore, 'nothing else reaches the Burrow'); - } finally { - await fixture.close(); - } -}); test('the transcript binds: a wrong prologue fails message 1', async () => { const fixture = await e2eFixture(); @@ -282,124 +236,6 @@ test('keepalives and control messages are one fixed size each', async () => { } }); -test('teardown: a closed Client socket tells the Burrow client-gone', async () => { - const fixture = await e2eFixture(); - const { burrow, client } = fixture; - const seen = watch(burrow); - try { - await client.open(); - await until(() => seen.opens.length === 1); - const { clientId } = seen.opens[0]; - - client.close(); - await until(() => burrow.frames.some((f) => f.t === 'client-gone' && f.clientId === clientId)); - assert.equal(burrow.e2eEntry(clientId), undefined, 'the ceremony went with the client'); - } finally { - await fixture.close(); - } -}); - -test('teardown: a replaced Burrow is burrow-gone and its late frames are dropped', async () => { - const fixture = await e2eFixture(); - const { burrow, client } = fixture; - const seen = watch(burrow); - try { - await client.open(); - await until(() => seen.opens.length === 1); - const entry = seen.opens[0]; - - const replacement = await fixture.replacementBurrow(); - const replaced = watch(replacement); - await client.waitFor((f) => f.t === 'burrow-gone'); - const closed = await burrow.closed; - assert.equal(closed.code, WS_CLOSE_BURROW_REPLACED); - - // The displaced socket speaks for nobody: the hub's map already points at - // the replacement, so a late transport frame is not forwarded. - burrow.e2eSendCiphertext(entry, entry.session.sendKeepalive()); - assert.equal(await client.quiet(), true); - - // The replacement is reachable, and its ceremonies are its own: a restarted - // Burrow has no memory of the session the Client held with its predecessor. - await client.open(); - await until(() => replaced.opens.length === 1); - assert.notDeepEqual(replaced.opens[0].session.handshakeHash, entry.session.handshakeHash); - } finally { - await fixture.close(); - } -}); - -test('a Burrow e2e frame for a Client bound elsewhere is not forwarded', async () => { - const fixture = await e2eFixture(); - const { burrow, client } = fixture; - const seen = watch(burrow); - try { - await client.open(); - await until(() => seen.opens.length === 1); - const entry = seen.opens[0]; - - // The Client rebinds to a different Burrow; the first one is told so. - const second = await fixture.secondBurrow(); - await client.open({ burrowId: second.burrowId }); - await until(() => burrow.frames.some((f) => f.t === 'client-gone')); - - burrow.e2eSendCiphertext(entry, entry.session.sendKeepalive()); - assert.equal(await client.quiet(), true, 'the old Burrow cannot reach the client'); - } finally { - await fixture.close(); - } -}); - -test('the relay is opaque: no plaintext, static, or handshake hash crosses it', async () => { - const fixture = await e2eFixture(); - const { burrow, client, burrowStatic, clientStatic } = fixture; - const MARKER = 'DORMOUSE-PLAINTEXT-ORACLE-9f3a'; - const opens = []; - burrow.on('e2e-open', (ev) => opens.push(ev)); - try { - await establish(fixture); - const seen = watch(burrow); - const entry = opens.at(-1); - - client.sendControl({ note: MARKER }); - client.sendApp(utf8Encode(`app ${MARKER}`)); - burrow.e2eSendApp(entry.clientId, utf8Encode(`reply ${MARKER}`)); - await until(() => seen.receipts.length === 2); - await client.nextTransport(); - - const view = relayView(client, burrow); - assert.equal(view.includes(MARKER), false, 'no plaintext crosses the relay'); - for (const [what, key] of [ - ['burrow static', burrowStatic.publicKey], - ['client static', clientStatic.publicKey], - ['handshake hash', client.session.handshakeHash], - ]) { - assert.equal(view.includes(toBase64Url(key)), false, `${what} must never appear`); - } - // What it *does* see is routing only. - assert.ok(view.includes(fixture.enrollment.burrowId)); - } finally { - await fixture.close(); - } -}); - -test('tampering with message 1 is rejected by the Burrow, and the relay cannot tell', async () => { - const fixture = await e2eFixture(); - const { burrow, client } = fixture; - const seen = watch(burrow); - try { - const id = newE2eId(); - await client.open({ id, tamper: (ct) => flip(ct), awaitResponse: false }); - await until(() => seen.errors.length === 1); - assert.equal(seen.opens.length, 0); - assert.equal(await client.quiet(), true); - // Forwarded, unexamined, exactly as the untampered one would have been. - assert.ok(burrow.frames.some((f) => f.t === 'e2e' && f.id === id && f.step === 'init')); - } finally { - await fixture.close(); - } -}); - test('tampering with message 2 is rejected by the Client', async () => { const fixture = await e2eFixture(); const { burrow, client } = fixture; @@ -439,110 +275,14 @@ test('tampering with a transport frame is rejected and poisons the session', asy } }); -test('the relay refuses malformed e2e frames before they reach the Burrow', async () => { - const fixture = await e2eFixture(); - const { burrow, client, enrollment } = fixture; - try { - const base = { - t: 'e2e', - burrowId: enrollment.burrowId, - kind: 'connection', - id: newE2eId(), - step: 'init', - ct: 'Zm9v', - }; - const before = burrow.frames.length; - const bad = [ - { ...base, ct: 'a'.repeat(MAX_E2E_CIPHERTEXT_LENGTH + 1) }, - { ...base, id: 'too-short' }, - { ...base, id: `${newE2eId()}x` }, - { ...base, kind: 'terminal' }, - { ...base, step: 'response' }, - { ...base, step: 'go' }, - { ...base, burrowId: 'not-a-burrow-id' }, - { ...base, ct: '' }, - ]; - for (const frame of bad) { - client.sendFrame(frame); - const error = await client.waitFor((f) => f.t === 'error'); - assert.equal(error.error, 'malformed e2e frame', JSON.stringify(frame)); - client.frames.length = 0; // consume, so the next wait sees a fresh one +// The routing cases every Relay passes, driven through this one. +for (const { name, run } of e2eCases) { + test(name, async () => { + const fixture = await e2eFixture(); + try { + await run(fixture); + } finally { + await fixture.close(); } - assert.equal(burrow.frames.length, before, 'nothing malformed reached the Burrow'); - - // A well-formed frame naming a Burrow that is not connected is the ordinary - // offline refusal, not a malformed one. - client.sendFrame({ ...base, burrowId: newE2eId() }); - const offline = await client.waitFor((f) => f.t === 'error'); - assert.match(offline.error, /is offline/); - } finally { - await fixture.close(); - } -}); - -test('a transport pipelined behind its init is handled after it, not beside it', async () => { - const fixture = await e2eFixture(); - const { burrow, client } = fixture; - const seen = watch(burrow); - try { - // Reading message 1 awaits three times before the session is recorded. A - // Burrow that handled socket frames concurrently would run this transport - // against a Map that does not hold the ceremony yet and answer "no e2e - // session" — the wrong diagnosis, and in stage 4 a dropped first payload. - const id = newE2eId(); - await client.open({ id, awaitResponse: false }); - client.sendCiphertext(toBase64Url(new Uint8Array(64)), { id }); - - await until(() => seen.errors.length === 1); - assert.equal(seen.opens.length, 1, 'the init completed first'); - assert.match( - String(seen.errors[0].error), - /authentication failed/, - 'the ceremony existed by the time its transport was read', - ); - } finally { - await fixture.close(); - } -}); - -test('a transport frame before any init is dropped, not forwarded', async () => { - const fixture = await e2eFixture(); - const { burrow, client, enrollment } = fixture; - try { - // A well-formed transport frame from a Client that has never bound: there - // is no binding to forward it within, so the relay drops it silently. - const before = burrow.frames.length; - client.sendFrame({ - t: 'e2e', - burrowId: enrollment.burrowId, - kind: 'connection', - id: newE2eId(), - step: 'transport', - ct: 'Zm9vYmFy', - }); - assert.equal(await client.quiet(), true, 'not even an error is answered'); - assert.equal(burrow.frames.length, before, 'transport never reaches an unbound Burrow'); - } finally { - await fixture.close(); - } -}); - -test('a transport frame outside the binding is dropped, not forwarded', async () => { - const fixture = await e2eFixture(); - const { burrow, client } = fixture; - const seen = watch(burrow); - try { - await client.open(); - await until(() => seen.opens.length === 1); - const second = await fixture.secondBurrow(); - - // A transport frame naming a Burrow this Client is not bound to. - const before = second.frames.length; - client.sendCiphertext(client.session.sendKeepalive(), {}); - client.sendFrame({ ...client.sent.at(-1), burrowId: second.burrowId }); - assert.equal(await client.quiet(), true); - assert.equal(second.frames.length, before, 'transport never binds a Burrow'); - } finally { - await fixture.close(); - } -}); + }); +} diff --git a/relay/test/harness/e2e.mjs b/relay/test/harness/e2e.mjs index 2bdf72a24..54991e781 100644 --- a/relay/test/harness/e2e.mjs +++ b/relay/test/harness/e2e.mjs @@ -1,65 +1,24 @@ /** - * The envelope facts every E2E test shares: which prologue a ceremony binds, - * and what a well-formed `e2e` frame looks like on each side of the relay. - * Shared so the fake Client and the fake Burrow cannot drift into two opinions - * about the transcript — a drift that would show up as a decrypt failure and - * read like a bug in the suite — and so a change to the envelope is one edit. + * The self-host Relay's E2E fixture. The envelope facts it shares with every + * Relay's suite — prologues, frame builders, `establish`, `watch`, `flip` — + * live in `remote-lib-common/test/harness/envelope.mjs`, re-exported here. */ -import assert from 'node:assert/strict'; -import { randomBytes } from 'node:crypto'; - -import { - E2E_ID_BYTE_LENGTH, - e2eConnectionPrologue, - e2ePairingPrologue, - fromBase64Url, - generateNoiseKeyPair, - toBase64Url, -} from 'remote-lib-common'; +import { generateNoiseKeyPair } from 'remote-lib-common'; import { enrollBurrow, freshApp, ownerSession, startRelay } from '../helpers.mjs'; -import { FakeClient } from './fake-client.mjs'; -import { FakeBurrow } from './fake-burrow.mjs'; +import { FakeClient } from '../../../remote-lib-common/test/harness/fake-client.mjs'; +import { FakeBurrow } from '../../../remote-lib-common/test/harness/fake-burrow.mjs'; -/** - * The prologue for one ceremony: the E2E version, the kind, the `burrowId`, and — - * for a connection — the connection id. - * - * The low-level door only: a real pairing binds every invitation field through - * `pairingInvitationPrologue`, which both halves of the harness call directly. - * The empty field list here is what a transcript-binding test wants — a - * prologue neither side's ceremony would ever build. - */ -export function e2ePrologueFor({ kind, burrowId, id }) { - return kind === 'connection' ? e2eConnectionPrologue(burrowId, id) : e2ePairingPrologue(burrowId, []); -} - -/** A fresh routing id, minted at the one length `isE2eId` accepts. */ -export function newE2eId() { - return toBase64Url(randomBytes(E2E_ID_BYTE_LENGTH)); -} - -/** - * A well-formed Client-originated `e2e` frame; the relay never decodes `ct`. - * `overrides` is how a test malforms exactly one field. - */ -export function e2eClientFrame(burrowId, overrides = {}) { - return { t: 'e2e', burrowId, kind: 'pairing', id: newE2eId(), step: 'init', ct: 'Zm9v', ...overrides }; -} - -/** Its Burrow-originated twin, addressed to `clientId` and carrying no `burrowId`. */ -export function e2eBurrowFrame(clientId, overrides = {}) { - return { - t: 'e2e', - clientId, - kind: 'pairing', - id: newE2eId(), - step: 'response', - ct: 'YmFy', - ...overrides, - }; -} +export { + e2eBurrowFrame, + e2eClientFrame, + e2ePrologueFor, + establish, + flip, + newE2eId, + watch, +} from '../../../remote-lib-common/test/harness/envelope.mjs'; /** * A live Relay, one Burrow with a Noise static, and one Client that pins it. @@ -69,7 +28,8 @@ export function e2eBurrowFrame(clientId, overrides = {}) { * the Relay's own relay. A factory because the relay binds the `burrowId` this * fixture only learns at enrollment. Shared so the honest and hostile suites * cannot drift into two fixtures — the difference between them has to be the - * relay and nothing else. + * relay and nothing else. Its shape is the one + * `remote-lib-common/test/harness/relay-parity.mjs` drives. */ export async function e2eFixture({ relayFor } = {}) { const created = await freshApp(); @@ -146,41 +106,3 @@ export async function e2eFixture({ relayFor } = {}) { }, }; } - -/** - * Pair and connect this fixture's Client, leaving an authorized session. - * - * The transport cases ride one, because that is where a Client's traffic - * actually lives: on a *pending* connection the Burrow answers the first control - * with an outcome and stops, exactly as `BurrowRuntime` does. - */ -export async function establish(fixture) { - const invitation = await fixture.burrow.mintInvitation(); - const paired = await fixture.client.pair({ - invitation, - authenticator: fixture.authenticator, - }); - assert.equal(paired.ok, true, JSON.stringify(paired.outcome)); - const connected = await fixture.client.connect({ authenticator: fixture.authenticator }); - assert.equal(connected.ok, true, JSON.stringify(connected.outcome)); - return connected; -} - -/** Record the Burrow's e2e outcomes so a test can await one. */ -export function watch(burrow) { - const receipts = []; - const errors = []; - const opens = []; - burrow.on('e2e-receive', (ev) => receipts.push(ev)); - burrow.on('e2e-error', (ev) => errors.push(ev)); - burrow.on('e2e-open', (ev) => opens.push(ev)); - return { receipts, errors, opens }; -} - -/** Flip one byte of a base64url ciphertext — what a hostile relay looks like. */ -export function flip(ct, index = -1) { - const bytes = fromBase64Url(ct); - const at = index < 0 ? bytes.length + index : index; - bytes[at] ^= 0x01; - return toBase64Url(bytes); -} diff --git a/relay/test/helpers.mjs b/relay/test/helpers.mjs index c783d1427..077c04e64 100644 --- a/relay/test/helpers.mjs +++ b/relay/test/helpers.mjs @@ -17,10 +17,12 @@ import { SimAuthenticator, registrationClientData as browserRegistrationClientData, } from '../../remote-lib-common/test/harness/actors.mjs'; +import { openFrameSocket } from '../../remote-lib-common/test/harness/frame-socket.mjs'; import { ORIGIN, PASSWORD, RP_ID } from './fixtures.mjs'; export * from './fixtures.mjs'; export { makeClock } from '../../remote-lib-common/test/harness/clock.mjs'; +export { sleep, until } from '../../remote-lib-common/test/harness/frame-socket.mjs'; /** * No app here pays the real `CREDENTIAL_FAILURE_DELAY_MS`: a suite full of 401s @@ -159,20 +161,6 @@ export async function signin(app, authenticator, { origin = ORIGIN, rpId = RP_ID // --- Slice 2: live Relay + WebSocket relay scaffolding -------------------- -export function sleep(ms) { - return new Promise((resolve) => setTimeout(resolve, ms)); -} - -/** Poll `fn` until it returns truthy, or throw after `timeout`ms. */ -export async function until(fn, { timeout = 1000, interval = 5 } = {}) { - const deadline = Date.now() + timeout; - for (;;) { - if (await fn()) return; - if (Date.now() > deadline) throw new Error('condition not met in time'); - await sleep(interval); - } -} - /** * Boot a real listening server for a `createApp` result (WS needs a socket, not * `app.request`). Binds port 0 and reports the OS-assigned port; the returned @@ -220,45 +208,12 @@ export function startRelay(created) { }); } -/** - * Open a WebSocket and wrap it in a tiny test harness: `ready` resolves on open - * (rejects on a failed upgrade), `take()` yields received frames in order with - * an internal cursor, and `quiet()` asserts no frame arrived in a window. - */ +/** {@link openFrameSocket}, tracked so a Relay teardown can force it shut. */ export function wsConnect(url) { - const ws = new WebSocket(url); - OPEN_SOCKETS.add(ws); - ws.addEventListener('close', () => OPEN_SOCKETS.delete(ws)); - const messages = []; - let cursor = 0; - ws.addEventListener('message', (ev) => { - messages.push(JSON.parse(typeof ev.data === 'string' ? ev.data : '')); - }); - const ready = new Promise((resolve, reject) => { - ws.addEventListener('open', () => resolve()); - ws.addEventListener('error', (ev) => reject(ev.error ?? new Error('ws error'))); - ws.addEventListener('close', (ev) => reject(new Error(`closed before open (${ev.code})`))); - }); - const closed = new Promise((resolve) => ws.addEventListener('close', (ev) => resolve(ev))); - return { - ws, - ready, - closed, - messages, - send: (frame) => ws.send(JSON.stringify(frame)), - close: () => ws.close(), - /** Next unconsumed frame, waiting up to `timeout`ms for it to arrive. */ - async take(timeout = 1000) { - await until(() => messages.length > cursor, { timeout }); - return messages[cursor++]; - }, - /** True if no new frame arrives within `ms` (i.e. the pipe stayed blocked). */ - async quiet(ms = 60) { - const before = messages.length; - await sleep(ms); - return messages.length === before; - }, - }; + const socket = openFrameSocket(url); + OPEN_SOCKETS.add(socket.ws); + socket.ws.addEventListener('close', () => OPEN_SOCKETS.delete(socket.ws)); + return socket; } /** POST /api/burrow/enroll with the setup password; returns the JSON body. */ diff --git a/relay/test/relay-limits.test.mjs b/relay/test/relay-limits.test.mjs index 3a2fbc812..343748257 100644 --- a/relay/test/relay-limits.test.mjs +++ b/relay/test/relay-limits.test.mjs @@ -1,103 +1,27 @@ /** * What bounds a relay socket (docs/specs/relay.md -> "Routing"). * - * The frame gates and routing rules live in `relay.test.mjs`; these are the - * resource bounds around them — how many Client sockets exist, how large a - * frame may be before any guard runs, and the two reasons a socket that passed - * the upgrade is closed afterwards. The upgrade check runs exactly once, which - * is why the sweep exists at all. + * The frame gates, routing rules, frame-size bound and Client cap are the + * cases every Relay passes (`relay.test.mjs`); these are the self-host + * sweep's: the two reasons a socket that passed the upgrade is closed + * afterwards. The upgrade check runs exactly once, which is why the sweep + * exists at all. */ import { test } from 'node:test'; import assert from 'node:assert/strict'; -import { WS_ROUTES, WS_TOKEN_PARAM } from 'remote-lib-common'; - -import { MAX_RELAY_FRAME_BYTES } from '../dist/app.js'; -import { MAX_RELAY_CLIENT_SOCKETS } from '../dist/relay.js'; - import { connectClient, connectBurrow, freshApp, makeClock, - ownerSession, startRelay, until, - wsConnect, } from './helpers.mjs'; import { e2eClientFrame, newE2eId } from './harness/e2e.mjs'; import { recordingSocket } from './harness/memory-socket.mjs'; -/** A live Relay plus a signed-in owner; the session token is reused per socket. */ -async function relayApp(options = {}) { - const created = await freshApp(options); - const server = await startRelay(created); - const { sessionToken } = await ownerSession(created.app); - const open = () => - wsConnect(`${server.wsUrl}${WS_ROUTES.client}?${WS_TOKEN_PARAM}=${sessionToken}`); - return { ...created, server, sessionToken, open }; -} - -test('client sockets are capped, and the refusal is a retry rather than an eviction', async (t) => { - const { hub, server, open } = await relayApp(); - t.after(() => server.close()); - - const sockets = []; - for (let i = 0; i < MAX_RELAY_CLIENT_SOCKETS; i += 1) { - const socket = open(); - await socket.ready; - sockets.push(socket); - } - assert.equal(hub.clientCount, MAX_RELAY_CLIENT_SOCKETS); - - // One past the cap: the upgrade succeeds (the session is valid) and the - // socket is closed immediately with "try again later". - const refused = open(); - const closed = await refused.closed; - assert.equal(closed.code, 1013); - // The sockets already relaying are untouched — dropping one to admit another - // would let a token holder take the relay away from itself. - assert.equal(hub.clientCount, MAX_RELAY_CLIENT_SOCKETS); - - for (const socket of sockets) socket.close(); -}); - -test('a frame larger than any legal one is refused by the socket, not buffered', async (t) => { - // `ws` defaults to 100 MiB, which the relay would buffer whole before - // `isE2eClientFrame` ran. 1009 is the protocol's own "message too big". - const { server, open } = await relayApp(); - t.after(() => server.close()); - - const socket = open(); - await socket.ready; - socket.ws.send('x'.repeat(MAX_RELAY_FRAME_BYTES + 1)); - - const closed = await socket.closed; - assert.equal(closed.code, 1009); -}); - -test('a maximal legal frame still fits under the cap', async (t) => { - const { server, open } = await relayApp(); - t.after(() => server.close()); - - const socket = open(); - await socket.ready; - // Well-formed but addressed to no live Burrow, so the answer is the routing - // error — which is the point: the frame was read rather than rejected by size. - socket.send({ - t: 'e2e', - burrowId: 'A'.repeat(22), - kind: 'connection', - id: 'B'.repeat(22), - step: 'init', - ct: 'C'.repeat(1024), - }); - const frame = await socket.take(); - assert.equal(frame.t, 'error'); - socket.close(); -}); - test('a client socket whose session has expired is closed by the sweep', async (t) => { // The upgrade gate runs once. A socket opened a minute before a 12-hour // session expires would otherwise relay for the process lifetime. diff --git a/relay/test/relay.test.mjs b/relay/test/relay.test.mjs index d92322ed7..4490019ec 100644 --- a/relay/test/relay.test.mjs +++ b/relay/test/relay.test.mjs @@ -1,246 +1,55 @@ /** - * Relay routing at the socket level (docs/specs/relay.md, "Relay"): two real - * in-process WebSockets echoing through the hub, with no ceremony behind them. + * Relay routing at the socket level (docs/specs/relay.md, "Routing"): real + * in-process WebSockets through the hub, with no ceremony behind them. * - * The relay routes exactly one envelope, so these cases are about the routing - * rules rather than about what rides inside: `clientId` stamping and stripping, - * the refusals, presence teardown (`client-gone` / `burrow-gone`), and burrow - * replacement. The envelope driven by real Noise ceremonies — including its - * bounds, the binding, and relay opacity — is `e2e-relay.test.mjs`. + * The cases are the ones every Relay passes, shared with Hosted's Durable + * Object suite in `remote-lib-common/test/harness/relay-parity.mjs`; this file + * is the self-host driver for them. The envelope driven by real Noise + * ceremonies is `e2e-relay.test.mjs`. */ import { test } from 'node:test'; -import assert from 'node:assert/strict'; -import { - WS_CLOSE_BURROW_REPLACED, - WS_CLOSE_BURROW_REPLACED_REASON, - WS_ROUTES, - WS_TOKEN_PARAM, -} from 'remote-lib-common'; +import { WS_ROUTES, WS_TOKEN_PARAM } from 'remote-lib-common'; -import { connectClient, connectBurrow, freshApp, startRelay, wsConnect } from './helpers.mjs'; -import { e2eClientFrame, newE2eId } from './harness/e2e.mjs'; +import { enrollBurrow, freshApp, ownerSession, startRelay, wsConnect } from './helpers.mjs'; +import { socketCases } from '../../remote-lib-common/test/harness/relay-parity.mjs'; -/** A boot-a-real-server fixture; every test tears its Relay down in `finally`. */ +/** A real server and the parity driver over it; every test tears its Relay down in `finally`. */ async function relay() { const created = await freshApp(); const server = await startRelay(created); - return { app: created.app, server, close: () => server.close() }; + const { sessionToken } = await ownerSession(created.app); + const burrowSocket = async (burrowToken) => { + const socket = wsConnect(`${server.wsUrl}${WS_ROUTES.burrow}?${WS_TOKEN_PARAM}=${burrowToken}`); + await socket.ready; + return socket; + }; + const openClient = () => + wsConnect(`${server.wsUrl}${WS_ROUTES.client}?${WS_TOKEN_PARAM}=${sessionToken}`); + const driver = { + async connectBurrow() { + const { body } = await enrollBurrow(created.app); + return { ...body, socket: await burrowSocket(body.burrowToken) }; + }, + reconnectBurrow: burrowSocket, + openClient, + async connectClient() { + const socket = openClient(); + await socket.ready; + return socket; + }, + }; + return { driver, close: () => server.close() }; } -test('an init round-trips client→burrow with a stamped clientId, and the answer routes back', async () => { - const { app, server, close } = await relay(); - try { - const { burrow, socket: burrowWs } = await connectBurrow(app, server); - const { socket: clientWs } = await connectClient(app, server); - - const sent = e2eClientFrame(burrow.burrowId); - clientWs.send(sent); - const forwarded = await burrowWs.take(); - assert.equal(forwarded.t, 'e2e'); - assert.equal(typeof forwarded.clientId, 'string'); - assert.equal(forwarded.id, sent.id); - assert.equal(forwarded.ct, sent.ct); - - burrowWs.send({ - t: 'e2e', - clientId: forwarded.clientId, - kind: 'pairing', - id: sent.id, - step: 'response', - ct: 'YmFy', - }); - const answer = await clientWs.take(); - assert.equal(answer.t, 'e2e'); - assert.equal(answer.burrowId, burrow.burrowId, 'the relay stamps the burrowId from the socket'); - assert.equal(answer.ct, 'YmFy'); - assert.equal(answer.clientId, undefined); // the clientId secret never leaks to the client - } finally { - await close(); - } -}); - -test('an e2e frame naming an offline burrow returns an error and nothing else', async () => { - const { app, server, close } = await relay(); - try { - const { socket: clientWs } = await connectClient(app, server); - clientWs.send(e2eClientFrame(newE2eId())); - const err = await clientWs.take(); - assert.equal(err.t, 'error'); - assert.match(err.error, /offline/); - assert.ok(await clientWs.quiet(), 'no further frames for an offline burrow'); - } finally { - await close(); - } -}); - -test('a transport outside a binding reaches no burrow, before and after one exists', async () => { - // The `init` is what binds; only it may create one. A `transport` that - // arrives with no binding — or naming a Burrow this socket has bound away - // from — has nowhere to go, and is dropped rather than answered, so nothing - // tells a prober which Burrows a session is talking to. - const { app, server, close } = await relay(); - try { - const a = await connectBurrow(app, server); - const b = await connectBurrow(app, server); - const { socket: clientWs } = await connectClient(app, server); - - // Never bound: the Burrow is online and the frame is well formed anyway. - clientWs.send(e2eClientFrame(a.burrow.burrowId, { step: 'transport' })); - assert.ok(await a.socket.quiet(), 'an unbound transport reaches no burrow'); - assert.ok(await clientWs.quiet(), 'and is dropped rather than answered'); - - // Bound to A, so a transport for B is outside the binding. - clientWs.send(e2eClientFrame(a.burrow.burrowId)); - assert.equal((await a.socket.take()).step, 'init'); - clientWs.send(e2eClientFrame(b.burrow.burrowId, { step: 'transport' })); - assert.ok(await b.socket.quiet(), 'a transport for the unbound burrow is dropped'); - assert.ok(await clientWs.quiet()); - - // The binding it does hold still carries. - clientWs.send(e2eClientFrame(a.burrow.burrowId, { step: 'transport' })); - assert.equal((await a.socket.take()).step, 'transport'); - } finally { - await close(); - } -}); - -test('malformed JSON and unknown client frames get an error; burrow garbage is ignored', async () => { - const { app, server, close } = await relay(); - try { - const { burrow, socket: burrowWs } = await connectBurrow(app, server); - const { socket: clientWs } = await connectClient(app, server); - - clientWs.ws.send('this is not json{'); - assert.equal((await clientWs.take()).t, 'error'); - - // Every frame the legacy handshake used is now exactly as unknown as any - // other word: the relay routes the `e2e` envelope and nothing else. - for (const t of ['pair', 'pair-status', 'connect', 'connect2', 'msg', 'nonsense-type']) { - clientWs.send({ t, burrowId: burrow.burrowId, data: {}, request: {} }); - const err = await clientWs.take(); - assert.equal(err.t, 'error'); - assert.equal(err.error, 'unknown frame type', t); +for (const { name, run } of socketCases) { + test(name, async () => { + const { driver, close } = await relay(); + try { + await run(driver); + } finally { + await close(); } - assert.ok(await burrowWs.quiet(), 'the burrow saw none of them'); - - // Garbage from the burrow is dropped without a reply or a crash — the relay - // still routes a following valid frame. - burrowWs.ws.send('garbage{'); - burrowWs.send({ t: 'unknown-burrow-frame', clientId: 'whatever' }); - assert.ok(await burrowWs.quiet()); - - clientWs.send(e2eClientFrame(burrow.burrowId)); - assert.equal((await burrowWs.take()).t, 'e2e'); - } finally { - await close(); - } -}); - -test('client disconnect delivers client-gone to its burrow', async () => { - const { app, server, close } = await relay(); - try { - const { burrow, socket: burrowWs } = await connectBurrow(app, server); - const { socket: clientWs } = await connectClient(app, server); - clientWs.send(e2eClientFrame(burrow.burrowId)); - const forwarded = await burrowWs.take(); - - clientWs.close(); - await clientWs.closed; - - const gone = await burrowWs.take(); - assert.deepEqual(gone, { t: 'client-gone', clientId: forwarded.clientId }); - } finally { - await close(); - } -}); - -test('binding to a second burrow tells the first the client is gone', async () => { - const { app, server, close } = await relay(); - try { - const a = await connectBurrow(app, server); - const b = await connectBurrow(app, server); - const { socket: clientWs } = await connectClient(app, server); - - clientWs.send(e2eClientFrame(a.burrow.burrowId)); - const first = await a.socket.take(); - clientWs.send(e2eClientFrame(b.burrow.burrowId)); - assert.equal((await b.socket.take()).t, 'e2e'); - assert.deepEqual(await a.socket.take(), { t: 'client-gone', clientId: first.clientId }); - } finally { - await close(); - } -}); - -test('burrow disconnect delivers burrow-gone to all its clients', async () => { - const { app, server, close } = await relay(); - try { - const { burrow, socket: burrowWs } = await connectBurrow(app, server); - const clientA = await connectClient(app, server); - const clientB = await connectClient(app, server); - clientA.socket.send(e2eClientFrame(burrow.burrowId)); - await burrowWs.take(); - clientB.socket.send(e2eClientFrame(burrow.burrowId)); - await burrowWs.take(); - - burrowWs.close(); - await burrowWs.closed; - - assert.deepEqual(await clientA.socket.take(), { t: 'burrow-gone' }); - assert.deepEqual(await clientB.socket.take(), { t: 'burrow-gone' }); - } finally { - await close(); - } -}); - -test('a burrow frame for a vanished client is dropped and the Relay keeps routing', async () => { - const { app, server, close } = await relay(); - try { - const { burrow, socket: burrowWs } = await connectBurrow(app, server); - const { socket: clientWs } = await connectClient(app, server); - clientWs.send(e2eClientFrame(burrow.burrowId)); - const forwarded = await burrowWs.take(); - - clientWs.close(); - await clientWs.closed; - await burrowWs.take(); // client-gone - - // The counterpart is gone; this must not throw or crash the process. - burrowWs.send({ ...forwarded, step: 'response', burrowId: undefined }); - - // Prove the relay is still alive: a fresh client still round-trips. - const client2 = await connectClient(app, server); - client2.socket.send(e2eClientFrame(burrow.burrowId)); - assert.equal((await burrowWs.take()).t, 'e2e'); - } finally { - await close(); - } -}); - -test('a new burrow socket replaces the old one for the same burrowId', async () => { - const { app, server, close } = await relay(); - try { - const first = await connectBurrow(app, server); - // Re-open /ws/burrow with the SAME token → same burrowId, displaces the first. - const second = wsConnect( - `${server.wsUrl}${WS_ROUTES.burrow}?${WS_TOKEN_PARAM}=${first.burrow.burrowToken}`, - ); - await second.ready; - - // The displaced socket is closed by the hub, carrying the code the evicted - // Burrow keys its stand-down on (lib/src/remote/burrow/burrow-runtime.ts). Pinned - // here because a changed code would silently restore the reconnect fight. - const closeEvent = await first.socket.closed; - assert.equal(closeEvent.code, WS_CLOSE_BURROW_REPLACED); - assert.equal(closeEvent.reason, WS_CLOSE_BURROW_REPLACED_REASON); - - // The new socket serves the same burrowId: a client's frame reaches it. - const { socket: clientWs } = await connectClient(app, server); - clientWs.send(e2eClientFrame(first.burrow.burrowId)); - assert.equal((await second.take()).t, 'e2e'); - second.close(); - } finally { - await close(); - } -}); + }); +} diff --git a/relay/test/setup-token.test.mjs b/relay/test/setup-token.test.mjs index 92851eeab..8e54fe727 100644 --- a/relay/test/setup-token.test.mjs +++ b/relay/test/setup-token.test.mjs @@ -24,7 +24,7 @@ import { import { AccountStore } from '../dist/state.js'; import { SetupTokenIssuer } from '../dist/setup-token.js'; -import { FakeBurrow } from './harness/fake-burrow.mjs'; +import { FakeBurrow } from '../../remote-lib-common/test/harness/fake-burrow.mjs'; import { ORIGIN, PASSWORD, diff --git a/remote-lib-common/src/index.ts b/remote-lib-common/src/index.ts index d0d5737a6..5c253fa9e 100644 --- a/remote-lib-common/src/index.ts +++ b/remote-lib-common/src/index.ts @@ -19,6 +19,7 @@ export * from './remote/enroll-offer.js'; export * from './remote/origin.js'; export * from './remote/enroll-code.js'; export * from './remote/relay-common.js'; +export * from './remote/relay-routing.js'; export * from './security/webcrypto.js'; export * from './security/bytes.js'; export * from './security/ecdsa.js'; diff --git a/remote-lib-common/src/remote/one-time-wire.ts b/remote-lib-common/src/remote/one-time-wire.ts index a8d5ac88d..274fdbf04 100644 --- a/remote-lib-common/src/remote/one-time-wire.ts +++ b/remote-lib-common/src/remote/one-time-wire.ts @@ -127,17 +127,6 @@ export function isOneTimeBurrowFrame(value: unknown): value is OneTimeBurrowFram ); } -/** - * The keepalive either end may send the room, and the room's answer. The room - * answers without waking and never forwards or counts either, and neither is - * JSON, so each is compared as a whole string before any parse. - */ -export const ONE_TIME_PING = 'ping'; -export const ONE_TIME_PONG = 'pong'; - -/** How often either end pings its rendezvous socket while it is open. */ -export const ONE_TIME_PING_INTERVAL_MS = 30_000; - // --------------------------------------------------------------------------- // Bounds and timings diff --git a/remote-lib-common/src/remote/relay-common.ts b/remote-lib-common/src/remote/relay-common.ts index b95001436..95b066836 100644 --- a/remote-lib-common/src/remote/relay-common.ts +++ b/remote-lib-common/src/remote/relay-common.ts @@ -24,6 +24,10 @@ import { import { boundedPushText } from '../security/push.js'; import { getWebCrypto } from '../security/webcrypto.js'; import { + E2E_ID_LENGTH, + MAX_CLIENT_ID_LENGTH, + MAX_E2E_CIPHERTEXT_LENGTH, + RELAY_PING_INTERVAL_MS, CLIENT_DATA_TYPE_ERROR, MALFORMED_ASSERTION_ERROR, MALFORMED_CLIENT_DATA_ERROR, @@ -38,6 +42,51 @@ import { /** The bearer shape lives in the wire contract; re-exported for the Relays. */ export { RELAY_BEARER_BYTE_LENGTH, RELAY_BEARER_LENGTH, isRelayBearer } from './wire.js'; +/** + * The largest relay frame either Relay will read, from either socket kind. + * Derived from the wire bounds the frame guards enforce — a maximal `ct` plus + * the envelope around it — so the raw text is bounded before any parse: the + * self-host Relay hands it to `ws` as `maxPayload` (which otherwise buffers up + * to 100 MiB), the Hosted Relay measures each message against it. + * `MAX_CLIENT_ID_LENGTH` is in here because a Burrow frame carries one. + */ +export const MAX_RELAY_FRAME_BYTES = + MAX_E2E_CIPHERTEXT_LENGTH + MAX_CLIENT_ID_LENGTH + 2 * E2E_ID_LENGTH + 1024; + +/** A frame over {@link MAX_RELAY_FRAME_BYTES} closes its socket with this. */ +export const WS_CLOSE_FRAME_TOO_LARGE = 1009; + +/** + * How many Client sockets one Relay holds at once: the self-host process, or + * one account's Durable Object on Hosted. One account's phones are a handful, + * so this is far above real use; without it a token-holder opens sockets until + * the Relay runs out. + */ +export const MAX_RELAY_CLIENT_SOCKETS = 64; + +/** Refused because the Relay is already holding {@link MAX_RELAY_CLIENT_SOCKETS}. */ +export const WS_CLOSE_TRY_AGAIN_LATER = 1013; + +/** + * The session behind this socket is gone. The same pair the `/ws/client` + * upgrade answers with, so a socket closed after the fact is indistinguishable + * from one refused at the door and Pocket needs no second recovery. + */ +export const WS_CLOSE_UNAUTHORIZED = 1008; +export const WS_CLOSE_UNAUTHORIZED_REASON = 'unauthorized'; + +/** A socket closed for silence, not for anything it did. */ +export const WS_CLOSE_IDLE = 1001; +export const WS_CLOSE_IDLE_REASON = 'no response to heartbeat'; + +/** + * How long either Relay lets a relay socket go unheard from before it counts + * as gone: three ping intervals. A live peer answers the first ping, so + * reaching this means the connection is half-open, not idle; generous against + * a phone whose radio has dozed, which reconnects anyway. + */ +export const RELAY_IDLE_TIMEOUT_MS = 3 * RELAY_PING_INTERVAL_MS; + /** Sessions live 12 hours (relay.md: "hours-scale TTL"). */ export const RELAY_SESSION_TTL_MS = 12 * 60 * 60 * 1000; diff --git a/remote-lib-common/src/remote/relay-routing.ts b/remote-lib-common/src/remote/relay-routing.ts new file mode 100644 index 000000000..17cea0f4b --- /dev/null +++ b/remote-lib-common/src/remote/relay-routing.ts @@ -0,0 +1,104 @@ +/** + * The frame layer both Relays route through (`docs/specs/relay.md` -> + * "Routing"): the self-host `RelayHub` (`relay/src/relay.ts`) and Hosted's + * `RelayRoom` (`hosted/server/relay-room.ts`). It reads the routing envelope + * through the shared guards and rebuilds it field by field, copying the + * ciphertext across unread. One copy, so the two Relays cannot answer a frame + * differently. + */ + +import { utf8Encode } from '../security/bytes.js'; +import { randomBase64Url } from '../security/webcrypto.js'; +import { MAX_RELAY_FRAME_BYTES } from './relay-common.js'; +import { + E2E_ID_BYTE_LENGTH, + isE2eBurrowFrame, + isE2eClientFrame, + type E2eBurrowFrame, + type E2eClientFrame, + type RelayToBurrowFrame, + type RelayToClientFrame, +} from './wire.js'; + +/** + * A Client socket's `clientId`: a Relay-assigned secret of 16 random bytes, + * stamped on every frame toward the Burrow and never sent to the Client. + */ +export const newClientId = (): string => randomBase64Url(E2E_ID_BYTE_LENGTH); + +/** The `error` a Client frame that is not a JSON object with a string `t` gets. */ +export const MALFORMED_FRAME_ERROR = 'malformed frame'; +/** The `error` a Client frame whose `t` is not `e2e` gets. */ +export const UNKNOWN_FRAME_TYPE_ERROR = 'unknown frame type'; +/** The `error` an `e2e` Client frame failing `isE2eClientFrame` gets. */ +export const MALFORMED_E2E_FRAME_ERROR = 'malformed e2e frame'; + +/** The `error` a Client frame naming a Burrow that holds no socket gets. */ +export const offlineError = (burrowId: string): RelayToClientFrame => ({ + t: 'error', + error: `burrow ${burrowId} is offline`, +}); + +/** + * Whether a received text frame is over {@link MAX_RELAY_FRAME_BYTES} in UTF-8 + * bytes, the unit `ws`'s `maxPayload` counts on the self-host Relay. A frame + * of at most a third of the bound in UTF-16 units is under it without encoding. + */ +export function exceedsRelayFrameBytes(raw: string): boolean { + if (raw.length * 3 <= MAX_RELAY_FRAME_BYTES) return false; + return raw.length > MAX_RELAY_FRAME_BYTES || utf8Encode(raw).byteLength > MAX_RELAY_FRAME_BYTES; +} + +/** A Client frame's routing envelope, or the `error` frame it is answered with. */ +export function readClientFrame( + raw: string, +): { frame: E2eClientFrame } | { error: RelayToClientFrame } { + const parsed = parseObject(raw); + if (!parsed || typeof parsed.t !== 'string') return { error: relayError(MALFORMED_FRAME_ERROR) }; + if (parsed.t !== 'e2e') return { error: relayError(UNKNOWN_FRAME_TYPE_ERROR) }; + if (!isE2eClientFrame(parsed)) return { error: relayError(MALFORMED_E2E_FRAME_ERROR) }; + return { frame: parsed }; +} + +/** A Burrow frame's routing envelope, or `null`: a malformed one is dropped unanswered. */ +export function readBurrowFrame(raw: string): E2eBurrowFrame | null { + const parsed = parseObject(raw); + return parsed && isE2eBurrowFrame(parsed) ? parsed : null; +} + +/** A Client's frame toward its Burrow, rebuilt field by field with the Relay's `clientId`. */ +export function toBurrowEnvelope(clientId: string, frame: E2eClientFrame): RelayToBurrowFrame { + return { + t: 'e2e', + clientId, + burrowId: frame.burrowId, + kind: frame.kind, + id: frame.id, + step: frame.step, + ct: frame.ct, + }; +} + +/** A Burrow's frame toward a Client, rebuilt field by field: `clientId` dropped, `burrowId` stamped. */ +export function toClientEnvelope(burrowId: string, frame: E2eBurrowFrame): RelayToClientFrame { + return { + t: 'e2e', + burrowId, + kind: frame.kind, + id: frame.id, + step: frame.step, + ct: frame.ct, + }; +} + +const relayError = (error: string): RelayToClientFrame => ({ t: 'error', error }); + +/** A raw text frame as a JSON object, or `null`. */ +function parseObject(raw: string): Record | null { + try { + const parsed: unknown = JSON.parse(raw); + return typeof parsed === 'object' && parsed !== null ? (parsed as Record) : null; + } catch { + return null; + } +} diff --git a/remote-lib-common/src/remote/wire.ts b/remote-lib-common/src/remote/wire.ts index c90fe2327..ca591fbb4 100644 --- a/remote-lib-common/src/remote/wire.ts +++ b/remote-lib-common/src/remote/wire.ts @@ -138,6 +138,26 @@ export const WS_ROUTES = { /** WS auth rides a query parameter (browsers cannot set WS headers). */ export const WS_TOKEN_PARAM = 'token'; +/** The 401 `GET /ws/burrow` answers a token that names no enrolled Burrow, on either Relay. */ +export const UNKNOWN_BURROW_TOKEN_ERROR = 'unknown burrow token'; + +/** + * The keepalive a Burrow or a Client sends its relay socket, and the Relay's + * answer (`docs/specs/relay.md` -> "Routing"); both ends of a one-time + * rendezvous send their room the same pair. Neither is JSON, so each is + * compared as a whole string before any parse; neither is forwarded. Hosted's + * Durable Objects answer without waking. + */ +export const RELAY_PING = 'ping'; +export const RELAY_PONG = 'pong'; + +/** + * How often either end pings its relay socket. Once a pong has arrived on a + * socket, a ping unanswered by the next one ends it; a Relay that never + * answers is never held to a deadline. + */ +export const RELAY_PING_INTERVAL_MS = 30_000; + /** * Close code the relay sends to a Burrow socket it displaces when a newer socket * claims the same `burrowId` (only one socket may own a burrowId — see relay.md diff --git a/remote-lib-common/test/harness/envelope.mjs b/remote-lib-common/test/harness/envelope.mjs new file mode 100644 index 000000000..c40853bad --- /dev/null +++ b/remote-lib-common/test/harness/envelope.mjs @@ -0,0 +1,102 @@ +/** + * The envelope facts every E2E test shares, whichever Relay it drives: which + * prologue a ceremony binds, what a well-formed `e2e` frame looks like on each + * side of the relay, and the pair-then-connect walk a transport case starts + * from. Shared so the fake Client and the fake Burrow cannot drift into two + * opinions about the transcript — a drift that would show up as a decrypt + * failure and read like a bug in the suite — and so a change to the envelope + * is one edit. + */ + +import assert from 'node:assert/strict'; +import { randomBytes } from 'node:crypto'; + +import { + E2E_ID_BYTE_LENGTH, + e2eConnectionPrologue, + e2ePairingPrologue, + fromBase64Url, + toBase64Url, +} from '../../dist/index.js'; + +/** + * The prologue for one ceremony: the E2E version, the kind, the `burrowId`, and — + * for a connection — the connection id. + * + * The low-level door only: a real pairing binds every invitation field through + * `pairingInvitationPrologue`, which both halves of the harness call directly. + * The empty field list here is what a transcript-binding test wants — a + * prologue neither side's ceremony would ever build. + */ +export function e2ePrologueFor({ kind, burrowId, id }) { + return kind === 'connection' ? e2eConnectionPrologue(burrowId, id) : e2ePairingPrologue(burrowId, []); +} + +/** A fresh routing id, minted at the one length `isE2eId` accepts. */ +export function newE2eId() { + return toBase64Url(randomBytes(E2E_ID_BYTE_LENGTH)); +} + +/** + * A well-formed Client-originated `e2e` frame; the relay never decodes `ct`. + * `overrides` is how a test malforms exactly one field. + */ +export function e2eClientFrame(burrowId, overrides = {}) { + return { t: 'e2e', burrowId, kind: 'pairing', id: newE2eId(), step: 'init', ct: 'Zm9v', ...overrides }; +} + +/** Its Burrow-originated twin, addressed to `clientId` and carrying no `burrowId`. */ +export function e2eBurrowFrame(clientId, overrides = {}) { + return { + t: 'e2e', + clientId, + kind: 'pairing', + id: newE2eId(), + step: 'response', + ct: 'YmFy', + ...overrides, + }; +} + +/** + * Pair and connect a fixture's Client, leaving an authorized session. The + * fixture names the account its Relay answers (`'owner'` on the self-host + * Relay, the user id on Hosted) as `accountId`, or leaves it to the Client's + * default. + * + * The transport cases ride one, because that is where a Client's traffic + * actually lives: on a *pending* connection the Burrow answers the first control + * with an outcome and stops, exactly as `BurrowRuntime` does. + */ +export async function establish(fixture) { + const invitation = await fixture.burrow.mintInvitation(); + const paired = await fixture.client.pair({ + invitation, + authenticator: fixture.authenticator, + ...(fixture.accountId !== undefined && { accountId: fixture.accountId }), + }); + assert.equal(paired.ok, true, JSON.stringify(paired.outcome)); + // The account rides in from the pairing outcome the Client kept. + const connected = await fixture.client.connect({ authenticator: fixture.authenticator }); + assert.equal(connected.ok, true, JSON.stringify(connected.outcome)); + return connected; +} + +/** Record the Burrow's e2e outcomes so a test can await one. */ +export function watch(burrow) { + const receipts = []; + const errors = []; + const opens = []; + burrow.on('e2e-receive', (ev) => receipts.push(ev)); + burrow.on('e2e-error', (ev) => errors.push(ev)); + burrow.on('e2e-open', (ev) => opens.push(ev)); + return { receipts, errors, opens }; +} + +/** Flip one byte of a base64url ciphertext — what a hostile relay looks like. */ +export function flip(ct, index = -1) { + const bytes = fromBase64Url(ct); + const at = index < 0 ? bytes.length + index : index; + bytes[at] ^= 0x01; + return toBase64Url(bytes); +} diff --git a/relay/test/harness/fake-burrow.mjs b/remote-lib-common/test/harness/fake-burrow.mjs similarity index 98% rename from relay/test/harness/fake-burrow.mjs rename to remote-lib-common/test/harness/fake-burrow.mjs index 43f4b6a60..54c5efec0 100644 --- a/relay/test/harness/fake-burrow.mjs +++ b/remote-lib-common/test/harness/fake-burrow.mjs @@ -10,10 +10,10 @@ * the same token) models a Burrow restart: its ACL starts empty again. * * Constructor: `{ relayUrl, burrowToken, burrowId, origin, rpId, label, - * autoApprove, requireUserVerification, noiseStaticKeyPair }`. `relayUrl` may - * be `http(s)://…` or `ws(s)://…`. With `autoApprove` the Burrow types back - * whatever code the request displayed; otherwise call - * `confirmPairing(clientId, code)` / `denyPairing(clientId)`. + * autoApprove, requireUserVerification, noiseStaticKeyPair, socket, + * socketInit }`. `relayUrl` may be `http(s)://…` or `ws(s)://…`. With + * `autoApprove` the Burrow types back whatever code the request displayed; + * otherwise call `confirmPairing(clientId, code)` / `denyPairing(clientId)`. * * Events, for logs and assertions: `open`, `close`, `frame`, `invitation`, * `e2e-open`, `e2e-receive`, `e2e-error`, `pairing-request`, `paired`, @@ -49,7 +49,7 @@ import { utf8Decode, utf8Encode, verifyPresenceProof, -} from 'remote-lib-common'; +} from '../../dist/index.js'; import { attachFrameSocket, closeSocket, receiveFrame, sendFrame } from './frame-socket.mjs'; @@ -73,6 +73,7 @@ export class FakeBurrow extends EventEmitter { requireUserVerification, noiseStaticKeyPair, socket, + socketInit, }) { super(); this.burrowId = burrowId; @@ -107,6 +108,7 @@ export class FakeBurrow extends EventEmitter { this, `${wsBase}${WS_ROUTES.burrow}?${WS_TOKEN_PARAM}=${burrowToken}`, socket, + socketInit, ); ws.addEventListener('message', (ev) => { // Serialized through a promise chain for the reason the relay serializes diff --git a/relay/test/harness/fake-client.mjs b/remote-lib-common/test/harness/fake-client.mjs similarity index 98% rename from relay/test/harness/fake-client.mjs rename to remote-lib-common/test/harness/fake-client.mjs index e05d05bfd..c7b5147af 100644 --- a/relay/test/harness/fake-client.mjs +++ b/remote-lib-common/test/harness/fake-client.mjs @@ -16,7 +16,7 @@ * low-level door for the transport cases in `relay/test/e2e-relay.test.mjs`. * * Constructor: `{ relayUrl, sessionToken, burrowId, staticKeyPair, - * burrowStaticPublicKey, origin, rpId, label }`. `relayUrl` may be + * burrowStaticPublicKey, origin, rpId, label, socket, socketInit }`. `relayUrl` may be * `http(s)://…` or `ws(s)://…`. Together with the Burrow's, this peer's `frames` * and `sent` are exactly what the relay saw, which is what the opacity * assertions read. @@ -40,9 +40,9 @@ import { toBase64Url, utf8Decode, utf8Encode, -} from 'remote-lib-common'; +} from '../../dist/index.js'; -import { e2ePrologueFor, newE2eId } from './e2e.mjs'; +import { e2ePrologueFor, newE2eId } from './envelope.mjs'; import { attachFrameSocket, closeSocket, @@ -63,6 +63,7 @@ export class FakeClient extends EventEmitter { rpId, label = 'Fake Phone', socket, + socketInit, }) { super(); this.burrowId = burrowId; @@ -96,6 +97,7 @@ export class FakeClient extends EventEmitter { this, `${wsBase}${WS_ROUTES.client}?${WS_TOKEN_PARAM}=${encodeURIComponent(sessionToken)}`, socket, + socketInit, ); ws.addEventListener('message', (ev) => receiveFrame(this, ev.data)); } diff --git a/relay/test/harness/frame-socket.mjs b/remote-lib-common/test/harness/frame-socket.mjs similarity index 57% rename from relay/test/harness/frame-socket.mjs rename to remote-lib-common/test/harness/frame-socket.mjs index b8e3f100b..de818dde0 100644 --- a/relay/test/harness/frame-socket.mjs +++ b/remote-lib-common/test/harness/frame-socket.mjs @@ -3,8 +3,9 @@ * frames in and out, and the `frames` / `sent` arrays every assertion reads. * * Shared so `FakeBurrow` and `FakeClient` cannot drift into two opinions about - * teardown or frame recording. It stays free of `relay/test/helpers.mjs` — - * `scripts/fake-burrow.mjs` imports `FakeBurrow` without a built Relay. + * teardown or frame recording. It stays free of any one Relay — the self-host + * suites, `relay/scripts/fake-burrow.mjs`, and Hosted's Durable Object suite + * all drive it. */ /** @@ -26,9 +27,11 @@ function record(log, frame) { * * `socket` replaces the real `WebSocket` — how the malicious-relay harness puts * a peer it controls between the two halves without changing either of them. + * `init` is Node's `WebSocket` options: `{ headers }` gives a socket the + * `Origin` a browser would send. */ -export function attachFrameSocket(target, url, socket) { - const ws = socket ?? new WebSocket(url); +export function attachFrameSocket(target, url, socket, init) { + const ws = socket ?? new WebSocket(url, init); target.ws = ws; /** Every frame the relay delivered, and every frame this peer sent. */ target.frames = []; @@ -108,3 +111,65 @@ export function closeSocket(target) { /* already closing */ } } + +export function sleep(ms) { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** Poll `fn` until it returns truthy, or throw after `timeout`ms. */ +export async function until(fn, { timeout = 1000, interval = 5 } = {}) { + const deadline = Date.now() + timeout; + for (;;) { + if (await fn()) return; + if (Date.now() > deadline) throw new Error('condition not met in time'); + await sleep(interval); + } +} + +/** + * Open a WebSocket and wrap it in a tiny test harness: `ready` resolves on open + * (rejects on a failed upgrade), `take()` yields received frames in order with + * an internal cursor — JSON parsed, anything else (a pong) as its text — and + * `quiet()` asserts no frame arrived in a window. `init` as + * {@link attachFrameSocket} takes it. + */ +export function openFrameSocket(url, init) { + const ws = new WebSocket(url, init); + const messages = []; + let cursor = 0; + ws.addEventListener('message', (ev) => { + const text = typeof ev.data === 'string' ? ev.data : ''; + try { + messages.push(JSON.parse(text)); + } catch { + messages.push(text); + } + }); + const ready = new Promise((resolve, reject) => { + ws.addEventListener('open', () => resolve()); + ws.addEventListener('error', (ev) => reject(ev.error ?? new Error('ws error'))); + ws.addEventListener('close', (ev) => reject(new Error(`closed before open (${ev.code})`))); + }); + // Unhandled when a test expects the upgrade to fail and awaits `closed` instead. + ready.catch(() => {}); + const closed = new Promise((resolve) => ws.addEventListener('close', (ev) => resolve(ev))); + return { + ws, + ready, + closed, + messages, + send: (frame) => ws.send(JSON.stringify(frame)), + close: () => ws.close(), + /** Next unconsumed frame, waiting up to `timeout`ms for it to arrive. */ + async take(timeout = 1000) { + await until(() => messages.length > cursor, { timeout }); + return messages[cursor++]; + }, + /** True if no new frame arrives within `ms` (i.e. the pipe stayed blocked). */ + async quiet(ms = 60) { + const before = messages.length; + await sleep(ms); + return messages.length === before; + }, + }; +} diff --git a/remote-lib-common/test/harness/relay-parity.mjs b/remote-lib-common/test/harness/relay-parity.mjs new file mode 100644 index 000000000..5700fcb71 --- /dev/null +++ b/remote-lib-common/test/harness/relay-parity.mjs @@ -0,0 +1,526 @@ +/** + * The routing cases every Relay passes (`docs/specs/relay.md` -> "Routing"), + * written once and registered by each Relay's suite: the self-host Relay's + * `relay/test/relay.test.mjs` and `relay/test/e2e-relay.test.mjs`, and + * Hosted's `hosted/server/tests/relay-room.test.ts`. One copy, so the two + * Relays cannot drift into two routings. + * + * {@link socketCases} drive bare sockets through a *driver*: + * + * - `connectBurrow()` → `{ burrowId, burrowToken, socket }`: a fresh enrolled + * Burrow of the account, its socket open. + * - `reconnectBurrow(burrowToken)` → a second open socket for that Burrow. + * - `connectClient()` → an open Client socket of the account. + * - `openClient()` → a Client socket, not awaited. + * + * Each socket is `openFrameSocket`'s (`./frame-socket.mjs`). + * + * {@link e2eCases} drive the real ceremonies through a fixture shaped like + * `relay/test/harness/e2e.mjs`'s `e2eFixture`: `{ burrow, client, + * authenticator, enrollment: { burrowId }, accountId?, replacementBurrow(), + * secondBurrow(), close() }`. + */ + +import assert from 'node:assert/strict'; + +import { + MAX_E2E_CIPHERTEXT_LENGTH, + MAX_RELAY_CLIENT_SOCKETS, + MAX_RELAY_FRAME_BYTES, + RELAY_PING, + RELAY_PONG, + REMOTE_METHODS, + WS_CLOSE_BURROW_REPLACED, + WS_CLOSE_BURROW_REPLACED_REASON, + WS_CLOSE_FRAME_TOO_LARGE, + WS_CLOSE_TRY_AGAIN_LATER, + toBase64Url, + utf8Encode, +} from '../../dist/index.js'; + +import { e2eClientFrame, establish, flip, newE2eId, watch } from './envelope.mjs'; +import { until } from './frame-socket.mjs'; + +/** One routing case: a name, and its body against a driver or fixture. */ +const relayCase = (name, run) => ({ name, run }); + +export const socketCases = [ + relayCase( + 'an init round-trips client→burrow with a stamped clientId, and the answer routes back', + async (relay) => { + const { burrowId, socket: burrowWs } = await relay.connectBurrow(); + const clientWs = await relay.connectClient(); + + const sent = e2eClientFrame(burrowId); + clientWs.send(sent); + const forwarded = await burrowWs.take(); + assert.equal(forwarded.t, 'e2e'); + assert.equal(typeof forwarded.clientId, 'string'); + assert.equal(forwarded.id, sent.id); + assert.equal(forwarded.ct, sent.ct); + + burrowWs.send({ + t: 'e2e', + clientId: forwarded.clientId, + kind: 'pairing', + id: sent.id, + step: 'response', + ct: 'YmFy', + }); + const answer = await clientWs.take(); + assert.equal(answer.t, 'e2e'); + assert.equal(answer.burrowId, burrowId, 'the relay stamps the burrowId from the socket'); + assert.equal(answer.ct, 'YmFy'); + assert.equal(answer.clientId, undefined); // the clientId secret never leaks to the client + }, + ), + + relayCase('an e2e frame naming an offline burrow returns an error and nothing else', async (relay) => { + const clientWs = await relay.connectClient(); + clientWs.send(e2eClientFrame(newE2eId())); + const err = await clientWs.take(); + assert.equal(err.t, 'error'); + assert.match(err.error, /offline/); + assert.ok(await clientWs.quiet(), 'no further frames for an offline burrow'); + }), + + relayCase('a transport outside a binding reaches no burrow, before and after one exists', async (relay) => { + // The `init` is what binds; only it may create one. A `transport` that + // arrives with no binding — or naming a Burrow this socket has bound away + // from — has nowhere to go, and is dropped rather than answered, so nothing + // tells a prober which Burrows a session is talking to. + const a = await relay.connectBurrow(); + const b = await relay.connectBurrow(); + const clientWs = await relay.connectClient(); + + // Never bound: the Burrow is online and the frame is well formed anyway. + clientWs.send(e2eClientFrame(a.burrowId, { step: 'transport' })); + assert.ok(await a.socket.quiet(), 'an unbound transport reaches no burrow'); + assert.ok(await clientWs.quiet(), 'and is dropped rather than answered'); + + // Bound to A, so a transport for B is outside the binding. + clientWs.send(e2eClientFrame(a.burrowId)); + assert.equal((await a.socket.take()).step, 'init'); + clientWs.send(e2eClientFrame(b.burrowId, { step: 'transport' })); + assert.ok(await b.socket.quiet(), 'a transport for the unbound burrow is dropped'); + assert.ok(await clientWs.quiet()); + + // The binding it does hold still carries. + clientWs.send(e2eClientFrame(a.burrowId, { step: 'transport' })); + assert.equal((await a.socket.take()).step, 'transport'); + }), + + relayCase('malformed JSON and unknown client frames get an error; burrow garbage is ignored', async (relay) => { + const { burrowId, socket: burrowWs } = await relay.connectBurrow(); + const clientWs = await relay.connectClient(); + + clientWs.ws.send('this is not json{'); + assert.equal((await clientWs.take()).t, 'error'); + + // Every frame the legacy handshake used is now exactly as unknown as any + // other word: the relay routes the `e2e` envelope and nothing else. + for (const t of ['pair', 'pair-status', 'connect', 'connect2', 'msg', 'nonsense-type']) { + clientWs.send({ t, burrowId, data: {}, request: {} }); + const err = await clientWs.take(); + assert.equal(err.t, 'error'); + assert.equal(err.error, 'unknown frame type', t); + } + assert.ok(await burrowWs.quiet(), 'the burrow saw none of them'); + + // Garbage from the burrow is dropped without a reply or a crash — the relay + // still routes a following valid frame. + burrowWs.ws.send('garbage{'); + burrowWs.send({ t: 'unknown-burrow-frame', clientId: 'whatever' }); + assert.ok(await burrowWs.quiet()); + + clientWs.send(e2eClientFrame(burrowId)); + assert.equal((await burrowWs.take()).t, 'e2e'); + }), + + relayCase('a ping is answered with a pong on either socket, and reaches no peer', async (relay) => { + const { burrowId, socket: burrowWs } = await relay.connectBurrow(); + const clientWs = await relay.connectClient(); + clientWs.send(e2eClientFrame(burrowId)); + await burrowWs.take(); + + clientWs.ws.send(RELAY_PING); + assert.equal(await clientWs.take(), RELAY_PONG); + burrowWs.ws.send(RELAY_PING); + assert.equal(await burrowWs.take(), RELAY_PONG); + // Neither is forwarded, and the Client's draws no `error`. + assert.ok(await burrowWs.quiet()); + assert.ok(await clientWs.quiet()); + }), + + relayCase('client disconnect delivers client-gone to its burrow', async (relay) => { + const { burrowId, socket: burrowWs } = await relay.connectBurrow(); + const clientWs = await relay.connectClient(); + clientWs.send(e2eClientFrame(burrowId)); + const forwarded = await burrowWs.take(); + + clientWs.close(); + await clientWs.closed; + + const gone = await burrowWs.take(); + assert.deepEqual(gone, { t: 'client-gone', clientId: forwarded.clientId }); + }), + + relayCase('binding to a second burrow tells the first the client is gone', async (relay) => { + const a = await relay.connectBurrow(); + const b = await relay.connectBurrow(); + const clientWs = await relay.connectClient(); + + clientWs.send(e2eClientFrame(a.burrowId)); + const first = await a.socket.take(); + clientWs.send(e2eClientFrame(b.burrowId)); + assert.equal((await b.socket.take()).t, 'e2e'); + assert.deepEqual(await a.socket.take(), { t: 'client-gone', clientId: first.clientId }); + }), + + relayCase('burrow disconnect delivers burrow-gone to all its clients', async (relay) => { + const { burrowId, socket: burrowWs } = await relay.connectBurrow(); + const clientA = await relay.connectClient(); + const clientB = await relay.connectClient(); + clientA.send(e2eClientFrame(burrowId)); + await burrowWs.take(); + clientB.send(e2eClientFrame(burrowId)); + await burrowWs.take(); + + burrowWs.close(); + await burrowWs.closed; + + assert.deepEqual(await clientA.take(), { t: 'burrow-gone' }); + assert.deepEqual(await clientB.take(), { t: 'burrow-gone' }); + }), + + relayCase('a burrow frame for a vanished client is dropped and the Relay keeps routing', async (relay) => { + const { burrowId, socket: burrowWs } = await relay.connectBurrow(); + const clientWs = await relay.connectClient(); + clientWs.send(e2eClientFrame(burrowId)); + const forwarded = await burrowWs.take(); + + clientWs.close(); + await clientWs.closed; + await burrowWs.take(); // client-gone + + // The counterpart is gone; this must not throw or crash the Relay. + burrowWs.send({ ...forwarded, step: 'response', burrowId: undefined }); + + // Prove the relay is still alive: a fresh client still round-trips. + const client2 = await relay.connectClient(); + client2.send(e2eClientFrame(burrowId)); + assert.equal((await burrowWs.take()).t, 'e2e'); + }), + + relayCase('a new burrow socket replaces the old one for the same burrowId', async (relay) => { + const first = await relay.connectBurrow(); + const clientWs = await relay.connectClient(); + clientWs.send(e2eClientFrame(first.burrowId)); + await first.socket.take(); + // Re-open the Burrow socket with the SAME token → same burrowId, displaces the first. + const second = await relay.reconnectBurrow(first.burrowToken); + + // The displaced socket is closed carrying the code the evicted Burrow keys + // its stand-down on (lib/src/remote/burrow/burrow-runtime.ts). Pinned here + // because a changed code would silently restore the reconnect fight. + const closeEvent = await first.socket.closed; + assert.equal(closeEvent.code, WS_CLOSE_BURROW_REPLACED); + assert.equal(closeEvent.reason, WS_CLOSE_BURROW_REPLACED_REASON); + // Its Client was told, and lost its binding at replacement time. + assert.deepEqual(await clientWs.take(), { t: 'burrow-gone' }); + clientWs.send(e2eClientFrame(first.burrowId, { step: 'transport' })); + assert.ok(await second.quiet(), 'the old binding does not carry to the replacement'); + + // The new socket serves the same burrowId: a client's frame reaches it. + const other = await relay.connectClient(); + other.send(e2eClientFrame(first.burrowId)); + assert.equal((await second.take()).t, 'e2e'); + }), + + relayCase('a frame larger than any legal one closes its socket 1009', async (relay) => { + for (const socket of [await relay.connectClient(), (await relay.connectBurrow()).socket]) { + socket.ws.send('x'.repeat(MAX_RELAY_FRAME_BYTES + 1)); + assert.equal((await socket.closed).code, WS_CLOSE_FRAME_TOO_LARGE); + } + }), + + relayCase('the frame bound counts UTF-8 bytes, not characters', async (relay) => { + // Two bytes a character: within the bound in characters, past it in bytes. + const text = 'é'.repeat(Math.floor(MAX_RELAY_FRAME_BYTES / 2) + 1); + assert.ok(text.length <= MAX_RELAY_FRAME_BYTES); + for (const socket of [await relay.connectClient(), (await relay.connectBurrow()).socket]) { + socket.ws.send(text); + assert.equal((await socket.closed).code, WS_CLOSE_FRAME_TOO_LARGE); + } + }), + + relayCase('a maximal legal frame is read, not refused by size', async (relay) => { + const socket = await relay.connectClient(); + // Well-formed but addressed to no live Burrow, so the answer is the routing + // error — which is the point: the frame was read rather than rejected by size. + const frame = e2eClientFrame(newE2eId(), { ct: 'A'.repeat(MAX_E2E_CIPHERTEXT_LENGTH) }); + assert.ok(JSON.stringify(frame).length <= MAX_RELAY_FRAME_BYTES); + socket.send(frame); + assert.match((await socket.take(5000)).error, /is offline/); + }), + + relayCase('client sockets are capped, and the refusal is a retry rather than an eviction', async (relay) => { + const { burrowId, socket: burrowWs } = await relay.connectBurrow(); + const sockets = []; + for (let i = 0; i < MAX_RELAY_CLIENT_SOCKETS; i += 1) sockets.push(await relay.connectClient()); + + // One past the cap: the upgrade succeeds (the session is valid) and the + // socket is closed at once with "try again later". + const refused = relay.openClient(); + assert.equal((await refused.closed).code, WS_CLOSE_TRY_AGAIN_LATER); + // The sockets already relaying are untouched — dropping one to admit + // another would let a token holder take the relay away from itself. + sockets[0].send(e2eClientFrame(burrowId)); + assert.equal((await burrowWs.take()).t, 'e2e'); + + // A closed one frees its slot. + sockets.pop().close(); + await until(async () => { + const retry = relay.openClient(); + const admitted = await Promise.race([ + retry.ready.then(() => retry.quiet(100)), + retry.closed.then(() => false), + ]); + if (!admitted) return false; + sockets.push(retry); + return true; + }, { timeout: 3000 }); + }), +]; + +/** Every frame the relay handled: what both peers sent and what it delivered. */ +function relayView(...peers) { + return JSON.stringify(peers.flatMap((peer) => [...peer.sent, ...peer.frames])); +} + +export const e2eCases = [ + relayCase('an established session round-trips every transport kind through the relay', async (fixture) => { + const { burrow, client } = fixture; + const opens = []; + burrow.on('e2e-open', (ev) => opens.push(ev)); + await establish(fixture); + // The connection handshake: both sides agree on the transcript, and IK + // authenticated the Client's static — the key the ACL conjunction matched. + const entry = opens.at(-1); + assert.deepEqual(entry.session.handshakeHash, client.session.handshakeHash); + assert.equal(entry.clientStaticPublicKey, toBase64Url(fixture.clientStatic.publicKey)); + + // Client → Burrow, all three kinds. + const seen = watch(burrow); + const payload = utf8Encode('terminal.write rides in here'); + client.sendKeepalive(); + client.sendControl({ presence: 'proof' }); + client.sendApp(payload); + await until(() => seen.receipts.length === 3); + assert.equal(seen.receipts[0].receipt.kind, 'keepalive'); + assert.deepEqual(seen.receipts[1].receipt, { kind: 'control', value: { presence: 'proof' } }); + assert.deepEqual(seen.receipts[2].receipt.messages, [payload]); + + // Burrow → Client, on the other direction's cipher state. + const reply = utf8Encode('terminal.data rides back'); + burrow.e2eSendApp(entry.clientId, reply); + const frame = await client.nextTransport(); + assert.equal(frame.burrowId, fixture.enrollment.burrowId, 'the relay stamps burrowId'); + assert.deepEqual(client.receiveFrame(frame).messages, [reply]); + + // protocol-v1 inside the same session. + const hello = await client.remoteRequest({ + requestId: 'r1', + method: REMOTE_METHODS.hello, + params: { protocolVersion: 1, viewer: 'phone' }, + }); + assert.equal(hello.ok, true); + assert.equal(hello.result.burrowId, fixture.enrollment.burrowId); + + // The envelope is the whole surface: an established session opens no other + // pipe, and every other frame type is simply unknown. + const burrowFramesBefore = burrow.frames.length; + client.sendFrame({ t: 'msg', data: { forbidden: true } }); + const refusal = await client.waitFor((f) => f.t === 'error'); + assert.equal(refusal.error, 'unknown frame type'); + assert.equal(burrow.frames.length, burrowFramesBefore, 'nothing else reaches the Burrow'); + }), + + relayCase('teardown: a closed Client socket tells the Burrow client-gone', async (fixture) => { + const { burrow, client } = fixture; + const seen = watch(burrow); + await client.open(); + await until(() => seen.opens.length === 1); + const { clientId } = seen.opens[0]; + + client.close(); + await until(() => burrow.frames.some((f) => f.t === 'client-gone' && f.clientId === clientId)); + assert.equal(burrow.e2eEntry(clientId), undefined, 'the ceremony went with the client'); + }), + + relayCase('teardown: a replaced Burrow is burrow-gone and its late frames are dropped', async (fixture) => { + const { burrow, client } = fixture; + const seen = watch(burrow); + await client.open(); + await until(() => seen.opens.length === 1); + const entry = seen.opens[0]; + + const replacement = await fixture.replacementBurrow(); + const replaced = watch(replacement); + await client.waitFor((f) => f.t === 'burrow-gone'); + const closed = await burrow.closed; + assert.equal(closed.code, WS_CLOSE_BURROW_REPLACED); + + // The displaced socket speaks for nobody, so a late transport frame is not forwarded. + burrow.e2eSendCiphertext(entry, entry.session.sendKeepalive()); + assert.equal(await client.quiet(), true); + + // The replacement is reachable, and its ceremonies are its own: a restarted + // Burrow has no memory of the session the Client held with its predecessor. + await client.open(); + await until(() => replaced.opens.length === 1); + assert.notDeepEqual(replaced.opens[0].session.handshakeHash, entry.session.handshakeHash); + }), + + relayCase('a Burrow e2e frame for a Client bound elsewhere is not forwarded', async (fixture) => { + const { burrow, client } = fixture; + const seen = watch(burrow); + await client.open(); + await until(() => seen.opens.length === 1); + const entry = seen.opens[0]; + + // The Client rebinds to a different Burrow; the first one is told so. + const second = await fixture.secondBurrow(); + await client.open({ burrowId: second.burrowId }); + await until(() => burrow.frames.some((f) => f.t === 'client-gone')); + + burrow.e2eSendCiphertext(entry, entry.session.sendKeepalive()); + assert.equal(await client.quiet(), true, 'the old Burrow cannot reach the client'); + }), + + relayCase('the relay is opaque: no plaintext, static, or handshake hash crosses it', async (fixture) => { + const { burrow, client } = fixture; + const MARKER = 'DORMOUSE-PLAINTEXT-ORACLE-9f3a'; + const opens = []; + burrow.on('e2e-open', (ev) => opens.push(ev)); + await establish(fixture); + const seen = watch(burrow); + const entry = opens.at(-1); + + client.sendControl({ note: MARKER }); + client.sendApp(utf8Encode(`app ${MARKER}`)); + burrow.e2eSendApp(entry.clientId, utf8Encode(`reply ${MARKER}`)); + await until(() => seen.receipts.length === 2); + await client.nextTransport(); + + const view = relayView(client, burrow); + assert.equal(view.includes(MARKER), false, 'no plaintext crosses the relay'); + for (const [what, key] of [ + ['burrow static', fixture.burrowStatic.publicKey], + ['client static', fixture.clientStatic.publicKey], + ['handshake hash', client.session.handshakeHash], + ]) { + assert.equal(view.includes(toBase64Url(key)), false, `${what} must never appear`); + } + // What it *does* see is routing only. + assert.ok(view.includes(fixture.enrollment.burrowId)); + }), + + relayCase('tampering with message 1 is forwarded unexamined and rejected by the Burrow', async (fixture) => { + const { burrow, client } = fixture; + const seen = watch(burrow); + const id = newE2eId(); + await client.open({ id, tamper: (ct) => flip(ct), awaitResponse: false }); + await until(() => seen.errors.length === 1); + assert.equal(seen.opens.length, 0); + assert.equal(await client.quiet(), true); + assert.ok(burrow.frames.some((f) => f.t === 'e2e' && f.id === id && f.step === 'init')); + }), + + relayCase('the relay refuses malformed e2e frames before they reach the Burrow', async (fixture) => { + const { burrow, client, enrollment } = fixture; + const base = { + t: 'e2e', + burrowId: enrollment.burrowId, + kind: 'connection', + id: newE2eId(), + step: 'init', + ct: 'Zm9v', + }; + const before = burrow.frames.length; + const bad = [ + { ...base, ct: 'a'.repeat(MAX_E2E_CIPHERTEXT_LENGTH + 1) }, + { ...base, id: 'too-short' }, + { ...base, id: `${newE2eId()}x` }, + { ...base, kind: 'terminal' }, + { ...base, step: 'response' }, + { ...base, step: 'go' }, + { ...base, burrowId: 'not-a-burrow-id' }, + { ...base, ct: '' }, + ]; + for (const frame of bad) { + client.sendFrame(frame); + const error = await client.waitFor((f) => f.t === 'error'); + assert.equal(error.error, 'malformed e2e frame', JSON.stringify(frame)); + client.frames.length = 0; // consume, so the next wait sees a fresh one + } + assert.equal(burrow.frames.length, before, 'nothing malformed reached the Burrow'); + + // A well-formed frame naming a Burrow that is not connected is the ordinary + // offline refusal, not a malformed one. + client.sendFrame({ ...base, burrowId: newE2eId() }); + const offline = await client.waitFor((f) => f.t === 'error'); + assert.match(offline.error, /is offline/); + }), + + relayCase('a transport pipelined behind its init is handled after it, not beside it', async (fixture) => { + const { burrow, client } = fixture; + const seen = watch(burrow); + // Reading message 1 awaits three times before the session is recorded. A + // Burrow that handled socket frames concurrently would run this transport + // against a Map that does not hold the ceremony yet. + const id = newE2eId(); + await client.open({ id, awaitResponse: false }); + client.sendCiphertext(toBase64Url(new Uint8Array(64)), { id }); + + await until(() => seen.errors.length === 1); + assert.equal(seen.opens.length, 1, 'the init completed first'); + assert.match( + String(seen.errors[0].error), + /authentication failed/, + 'the ceremony existed by the time its transport was read', + ); + }), + + relayCase('a transport frame before any init is dropped, not forwarded', async (fixture) => { + const { burrow, client, enrollment } = fixture; + // A well-formed transport frame from a Client that has never bound: there + // is no binding to forward it within, so the relay drops it silently. + const before = burrow.frames.length; + client.sendFrame({ + t: 'e2e', + burrowId: enrollment.burrowId, + kind: 'connection', + id: newE2eId(), + step: 'transport', + ct: 'Zm9vYmFy', + }); + assert.equal(await client.quiet(), true, 'not even an error is answered'); + assert.equal(burrow.frames.length, before, 'transport never reaches an unbound Burrow'); + }), + + relayCase('a transport frame outside the binding is dropped, not forwarded', async (fixture) => { + const { burrow, client } = fixture; + const seen = watch(burrow); + await client.open(); + await until(() => seen.opens.length === 1); + const second = await fixture.secondBurrow(); + + // A transport frame naming a Burrow this Client is not bound to. + const before = second.frames.length; + client.sendCiphertext(client.session.sendKeepalive(), {}); + client.sendFrame({ ...client.sent.at(-1), burrowId: second.burrowId }); + assert.equal(await client.quiet(), true); + assert.equal(second.frames.length, before, 'transport never binds a Burrow'); + }), +]; diff --git a/remote-lib-common/test/one-time-wire.test.mjs b/remote-lib-common/test/one-time-wire.test.mjs index 6614b9b8e..a847aab84 100644 --- a/remote-lib-common/test/one-time-wire.test.mjs +++ b/remote-lib-common/test/one-time-wire.test.mjs @@ -17,9 +17,9 @@ import { ONE_TIME_DIRECT_DEADLINE_MS, ONE_TIME_EXPIRY_GRACE_MS, ONE_TIME_LINK_TTL_MS, - ONE_TIME_PING, - ONE_TIME_PING_INTERVAL_MS, - ONE_TIME_PONG, + RELAY_PING, + RELAY_PING_INTERVAL_MS, + RELAY_PONG, ONE_TIME_ROOM_PARAM, ONE_TIME_WS_ROUTES, WS_CLOSE_BURROW_REPLACED, @@ -179,12 +179,12 @@ test('the timings: a pairing-length link, a short grace, and a direct deadline i assert.ok(ONE_TIME_DIRECT_DEADLINE_MS < ONE_TIME_EXPIRY_GRACE_MS); }); -test('the keepalive is two fixed strings no frame can be, on a shared interval', () => { - assert.equal(ONE_TIME_PING, 'ping'); - assert.equal(ONE_TIME_PONG, 'pong'); - assert.equal(ONE_TIME_PING_INTERVAL_MS, 30_000); +test('the rendezvous keepalive is the relay socket\'s: two fixed strings no frame can be', () => { + assert.equal(RELAY_PING, 'ping'); + assert.equal(RELAY_PONG, 'pong'); + assert.equal(RELAY_PING_INTERVAL_MS, 30_000); // Neither parses as a frame, so neither can be mistaken for one. - for (const text of [ONE_TIME_PING, ONE_TIME_PONG]) { + for (const text of [RELAY_PING, RELAY_PONG]) { assert.throws(() => JSON.parse(text)); } }); diff --git a/remote-lib-common/test/relay-routing.test.mjs b/remote-lib-common/test/relay-routing.test.mjs new file mode 100644 index 000000000..1875e98af --- /dev/null +++ b/remote-lib-common/test/relay-routing.test.mjs @@ -0,0 +1,67 @@ +/** + * The frame layer both Relays route through (docs/specs/relay.md -> Routing): + * the byte bound, the Client frame's three refusals, the dropped Burrow frame, + * and the envelopes rebuilt field by field. The Relays' own suites run the + * same rules end to end through `test/harness/relay-parity.mjs`. + */ + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { + MALFORMED_E2E_FRAME_ERROR, + MALFORMED_FRAME_ERROR, + MAX_RELAY_FRAME_BYTES, + UNKNOWN_FRAME_TYPE_ERROR, + exceedsRelayFrameBytes, + offlineError, + readBurrowFrame, + readClientFrame, + toBurrowEnvelope, + toClientEnvelope, +} from '../dist/index.js'; + +const ID = 'AAAAAAAAAAAAAAAAAAAAAA'; +const clientFrame = { t: 'e2e', burrowId: ID, kind: 'pairing', id: ID, step: 'init', ct: 'Zm9v' }; +const burrowFrame = { t: 'e2e', clientId: 'c1', kind: 'pairing', id: ID, step: 'response', ct: 'YmFy' }; + +test('the frame bound counts UTF-8 bytes, as `ws` maxPayload does', () => { + assert.equal(exceedsRelayFrameBytes('x'.repeat(MAX_RELAY_FRAME_BYTES)), false); + assert.equal(exceedsRelayFrameBytes('x'.repeat(MAX_RELAY_FRAME_BYTES + 1)), true); + // Two bytes a code unit: under the bound in UTF-16 units, over it in bytes. + const twoByte = 'é'.repeat(Math.floor(MAX_RELAY_FRAME_BYTES / 2) + 1); + assert.ok(twoByte.length <= MAX_RELAY_FRAME_BYTES); + assert.equal(exceedsRelayFrameBytes(twoByte), true); + assert.equal(exceedsRelayFrameBytes('é'.repeat(Math.floor(MAX_RELAY_FRAME_BYTES / 2))), false); + // Four bytes a surrogate pair, two code units. + const astral = '😀'.repeat(Math.floor(MAX_RELAY_FRAME_BYTES / 4) + 1); + assert.equal(exceedsRelayFrameBytes(astral), true); + assert.equal(exceedsRelayFrameBytes('😀'.repeat(Math.floor(MAX_RELAY_FRAME_BYTES / 4))), false); +}); + +test('a Client frame is the e2e envelope, or one of three errors', () => { + assert.deepEqual(readClientFrame(JSON.stringify(clientFrame)), { frame: clientFrame }); + for (const raw of ['not json', '42', 'null', '{}', '{"t":1}']) + assert.deepEqual(readClientFrame(raw), { error: { t: 'error', error: MALFORMED_FRAME_ERROR } }, raw); + assert.deepEqual(readClientFrame('{"t":"hello"}'), { + error: { t: 'error', error: UNKNOWN_FRAME_TYPE_ERROR }, + }); + assert.deepEqual(readClientFrame(JSON.stringify({ ...clientFrame, ct: '' })), { + error: { t: 'error', error: MALFORMED_E2E_FRAME_ERROR }, + }); + assert.deepEqual(offlineError(ID), { t: 'error', error: `burrow ${ID} is offline` }); +}); + +test('a malformed Burrow frame is dropped, not answered', () => { + assert.deepEqual(readBurrowFrame(JSON.stringify(burrowFrame)), burrowFrame); + for (const raw of ['not json', '{}', JSON.stringify({ ...burrowFrame, step: 'init' })]) + assert.equal(readBurrowFrame(raw), null, raw); +}); + +test('each envelope is rebuilt field by field: nothing a sender added rides along', () => { + const toBurrow = toBurrowEnvelope('relay-id', { ...clientFrame, clientId: 'forged', extra: 1 }); + assert.deepEqual(toBurrow, { ...clientFrame, clientId: 'relay-id' }); + const toClient = toClientEnvelope(ID, { ...burrowFrame, burrowId: 'forged', extra: 1 }); + const { clientId: _dropped, ...rest } = burrowFrame; + assert.deepEqual(toClient, { ...rest, burrowId: ID }); +}); diff --git a/scripts/e2e-lint-selftest.mjs b/scripts/e2e-lint-selftest.mjs index 9c764f766..8477c3b45 100644 --- a/scripts/e2e-lint-selftest.mjs +++ b/scripts/e2e-lint-selftest.mjs @@ -34,6 +34,8 @@ import { ICE_SERVER_MODULE, NATIVE_PEER_FACTORY, PEER_FACTORIES, + RELAY_ROOM, + RELAY_ROUTING, RULES, SECURITY_SPEC, } from './e2e-lint.mjs'; @@ -272,6 +274,54 @@ for (const violation of [ ); } +// The same for the per-account relay object, which reaches frames only through +// the shared frame layer: every way of naming, reading, logging, or keeping one +// must redden, including the four a review found past the first version of the +// rule — a `Buffer` decode, a `TextDecoder`, a destructured `ct`, and a frame +// spread into an attachment. +for (const violation of [ + '\nconst __selftest = (frame: { ct: string }) => Buffer.from(frame.ct, "base64");\n', + '\nconst __selftest = (bytes: Uint8Array) => new TextDecoder().decode(bytes);\n', + '\nconst __selftest = (frame: object) => { const { ct } = frame as { ct: string }; return ct; };\n', + '\nconst __selftest = (ws: WorkerWebSocket, conn: object, raw: string) => ws.serializeAttachment({ ...conn, last: raw });\n', + '\nconst __selftest = (ws: WorkerWebSocket, raw: string) => ws.serializeAttachment({ role: "client", raw });\n', + '\nconst __selftest = (conn: { last?: string }, raw: string) => { conn.last = raw; };\n', + '\nconst __selftest = (frame: Record) => frame["ct"];\n', + '\nconst __selftest = (raw: string) => JSON.parse(raw);\n', + '\nconst __selftest = (raw: string) => atob(raw);\n', + '\nconst __selftest = (raw: string) => fromBase64Url(raw);\n', + '\nconst __selftest = (frame: string) => console.log(frame);\n', + '\nconst __selftest = (frame: string) => console.error("RelayRoom refused a request for another account", frame);\n', + '\nconst __selftest = (frame: string) => console.error(`refused ${frame}`);\n', + '\nconst __selftest = console.error;\n', + "\nconst __selftest = (frame: string) => this.ctx.storage.put('frame', frame);\n", + '\nconst __selftest = (frame: string) => this.ctx.storage.put(ACCOUNT_KEY, frame);\n', + "\nconst __selftest = (frame: string) => this.ctx.storage.sql.exec('SELECT ?', frame);\n", + '\nconst __selftest = (ctx: DurableObjectState) => ctx.storage;\n', +]) { + selftest.withAppended( + RELAY_ROOM, + violation, + `a forbidden read or write in ${RELAY_ROOM} stays green: ${violation.trim()}`, + ); +} + +// The shared frame layer may copy the ciphertext and nothing else. +for (const violation of [ + '\nconst __selftest = (frame: { ct: string }) => { const { ct } = frame; return ct; };\n', + '\nconst __selftest = (frame: { ct: string }) => frame.ct.length;\n', + '\nconst __selftest = (frame: { ct: string }) => fromBase64Url(frame.ct);\n', + '\nconst __selftest = (bytes: Uint8Array) => new TextDecoder().decode(bytes);\n', + '\nconst __selftest = (text: string) => JSON.parse(text);\n', + '\nconst __selftest = (raw: string) => console.log(raw);\n', +]) { + selftest.withAppended( + RELAY_ROUTING, + violation, + `a forbidden read in ${RELAY_ROUTING} stays green: ${violation.trim()}`, + ); +} + // A file-scoped storage exception must not become a directory-scoped escape. selftest.withAppended( 'lib/src/remote/client/pocket-db.ts', diff --git a/scripts/e2e-lint.mjs b/scripts/e2e-lint.mjs index 0f30e3b73..6745ac93c 100644 --- a/scripts/e2e-lint.mjs +++ b/scripts/e2e-lint.mjs @@ -1,9 +1,9 @@ #!/usr/bin/env node /** * Mechanical check for the structural half of the end-to-end boundary in - * `docs/specs/security-remote.md` ("Remote Control"), and of the Hosted room - * that forwards a one-time handshake in `docs/specs/security-hosted.md` - * ("Rendezvous boundary"). Runs from the repo + * `docs/specs/security-remote.md` ("Remote Control"), and of the Hosted rooms + * that forward a one-time handshake and an account's relay frames in + * `docs/specs/security-hosted.md` ("Rendezvous boundary", "Relay boundary"). Runs from the repo * root via `pnpm test` (see the root package.json). Exits non-zero with a * per-violation report naming the rule that was broken and the spec line it * enforces. @@ -64,7 +64,7 @@ const NOISE_PROTOCOL_NAME = 'Noise_IK_25519_ChaChaPoly_SHA256'; /** The spec whose "Remote Control" lines the rules below pin, unless a rule names its own. */ export const SECURITY_SPEC = 'docs/specs/security-remote.md'; -/** The spec whose "Rendezvous boundary" lines the Hosted room's rule pins. */ +/** The spec whose "Rendezvous boundary" and "Relay boundary" lines the Hosted rooms' rules pin. */ export const HOSTED_SECURITY_SPEC = 'docs/specs/security-hosted.md'; /** @@ -178,6 +178,36 @@ const GRANT_NAME = */ const ONE_TIME_NAME = /[Oo]neTime|ONE_TIME_|['"`]one-time/g; +/** Hosted's per-account relay object (`docs/specs/hosted.md` -> "Relay sockets"). */ +export const RELAY_ROOM = 'hosted/server/relay-room.ts'; + +/** The frame layer both Relays route through (`docs/specs/relay.md` -> "Routing"). */ +export const RELAY_ROUTING = 'remote-lib-common/src/remote/relay-routing.ts'; + +/** + * Everything {@link RELAY_ROOM} could keep, log, or read a frame through, + * spelled out because it reaches frames only through {@link RELAY_ROUTING}: + * + * - the ciphertext's field name at all, a parse, or a decode — it never + * names `ct`, so a destructure or a computed key is a match too; + * - `console` other than one method called with one plain string; + * - `storage` other than the reads, the alarm, and the one write of the + * account id — an alias of it included; + * - an attachment other than a connection (`conn`, `x.conn`) or a connection + * literal checked `satisfies` its type without a spread, and a connection + * field written other than the two routing writes. + */ +const RELAY_ROOM_LEAKS = new RegExp( + [ + String.raw`\bct\b|\bJSON\.parse\b|\batob\b|\bBuffer\b|\bTextDecoder\b|[Bb]ase64`, + String.raw`\bconsole\b(?!\.\w+\(\s*"[^"\\]*"\s*\))`, + String.raw`\bstorage\b(?!\.(?:get|getAlarm|setAlarm|deleteAlarm)\b|\.put\(ACCOUNT_KEY, account\))`, + String.raw`\bserializeAttachment\((?!(?:\w+\.)?conn\)|\{(?:(?!\.\.\.)[^{}()])*\}\s*satisfies\s+(?:BurrowConn|ClientConn)\))`, + String.raw`\bconn\.(?!retired\b|burrowId\b)\w+\s*=(?!=)`, + ].join('|'), + 'g', +); + /** The three shipped source trees, scanned whole for the dependency rules. */ const SOURCE_TREES = ['remote-lib-common/src/', 'lib/src/', 'relay/src/']; @@ -428,6 +458,33 @@ export const RULES = [ violationFile: 'hosted/server/one-time-room.ts', violation: '\nconst __selftest = (frame: string) => JSON.parse(frame);\n', }, + { + rule: "Hosted's RelayRoom never names, parses, decodes, logs, or stores a frame", + spec: HOSTED_SECURITY_SPEC, + security: 'stores, logs, or decodes a frame or its `ct`', + kind: 'forbid', + files: [RELAY_ROOM], + // It reads a frame only through the shared frame layer, which hands back + // the routing envelope and rebuilds it; everything that could keep, log, + // or read one is in `RELAY_ROOM_LEAKS`. + pattern: RELAY_ROOM_LEAKS, + violationFile: RELAY_ROOM, + violation: '\nconst __selftest = (frame: { ct: string }) => Buffer.from(frame.ct, "base64");\n', + }, + { + rule: 'The shared frame layer copies `ct` field by field and reads it nowhere', + spec: HOSTED_SECURITY_SPEC, + security: 'copies `ct` field by field and reads it nowhere else', + kind: 'forbid', + files: [RELAY_ROUTING], + // Its one parse is of the raw frame, after which the guards bound every + // field; `ct` appears only as the copy `ct: frame.ct,` into an envelope. + pattern: + /\bct: frame\.ct,|\bct\b|\bJSON\.parse\b(?!\(raw\))|\batob\b|\bBuffer\b|\bTextDecoder\b|\bfromBase64Url\b|\bconsole\b/g, + allow: (match) => match === 'ct: frame.ct,', + violationFile: RELAY_ROUTING, + violation: '\nconst __selftest = (frame: { ct: string }) => atob(frame.ct);\n', + }, { rule: 'The Relay never names the one-time family', security: 'no one-time name may appear under `relay/src/`', @@ -685,7 +742,7 @@ if (import.meta.url === pathToFileURL(process.argv[1]).href) { for (const failure of failures) console.error(` ${failure}\n`); console.error( `Each line above maps to the "Remote Control" section of ${SECURITY_SPEC}, or to the\n` + - `"Rendezvous boundary" section of ${HOSTED_SECURITY_SPEC}. If a\n` + + `"Rendezvous boundary" or "Relay boundary" section of ${HOSTED_SECURITY_SPEC}. If a\n` + 'control moved rather than disappeared, update the rule in scripts/e2e-lint.mjs\n' + 'in the same commit — and add the self-test case that proves it load-bearing.', ); diff --git a/scripts/spec-word-budgets.json b/scripts/spec-word-budgets.json index 30d308bdb..0f8018555 100644 --- a/scripts/spec-word-budgets.json +++ b/scripts/spec-word-budgets.json @@ -1,5 +1,5 @@ { - "AGENTS.md": 3550, + "AGENTS.md": 3600, "SECURITY.md": 200, "SELF_HOST.md": 6100, "docs/compatible-agents.md": 1750, @@ -12,19 +12,19 @@ "docs/specs/dor-tools-builtin.md": 1050, "docs/specs/dor-tools-lib.md": 400, "docs/specs/glossary.md": 3000, - "docs/specs/hosted.md": 3550, + "docs/specs/hosted.md": 4250, "docs/specs/layout.md": 11500, "docs/specs/mobile-terminal-ui.md": 2300, "docs/specs/mouse-and-clipboard.md": 5250, "docs/specs/one-time.md": 3850, "docs/specs/pocket-app.md": 5100, - "docs/specs/relay.md": 10100, + "docs/specs/relay.md": 10200, "docs/specs/remote-api.md": 5250, "docs/specs/remote-network.md": 2450, "docs/specs/remote-security-model.md": 5450, "docs/specs/security-audit.md": 2100, "docs/specs/security-ci.md": 2950, - "docs/specs/security-hosted.md": 1900, + "docs/specs/security-hosted.md": 2150, "docs/specs/security-local.md": 3950, "docs/specs/security-remote.md": 7050, "docs/specs/security-supply-chain.md": 1250,