diff --git a/.changeset/admin-users-batched-connection-read.md b/.changeset/admin-users-batched-connection-read.md deleted file mode 100644 index ff58bd052..000000000 --- a/.changeset/admin-users-batched-connection-read.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -"@executor-js/sdk": patch ---- - -**Fix: the admin joined user view no longer issues one connection query per subject** - -`admin.listSubjectsWithConnections` read a page of subjects and then queried -connections once per subject, sequentially. A default page therefore cost 100 -round trips inside a single request, which on a per-request socket dominated -the response. It now reads the page and then batches every subject's -connections into one query, so the cost is two queries regardless of page size. -A subject with no connections still reports an empty array rather than dropping -out of the page, and the batched read carries the same `owner: "user"` and -tenant scoping the per-subject read did. - -The `?email=` filter on the admin users endpoints is also applied before the -read rather than after it: the address resolves to a principal id and that id is -read directly, instead of paging the tenant and keeping the row that matched. -Paging still applies to a filtered response, but to the selected row — one row -at `offset: 0`, empty beyond it. diff --git a/.changeset/graph-slice-urls-first-class.md b/.changeset/graph-slice-urls-first-class.md new file mode 100644 index 000000000..fd2ba2e36 --- /dev/null +++ b/.changeset/graph-slice-urls-first-class.md @@ -0,0 +1,5 @@ +--- +"@executor-js/plugin-openapi": patch +--- + +Make Microsoft Graph slice URLs first-class spec sources instead of a hidden substitution. Catalog tiles now point directly at the slice release assets, the stored specUrl is exactly what gets fetched, and selection narrowing travels visibly in the URL fragment; requesting the upstream monolith URL fetches the monolith, never a silently swapped slice. diff --git a/.changeset/graph-spec-slices.md b/.changeset/graph-spec-slices.md new file mode 100644 index 000000000..16345d752 --- /dev/null +++ b/.changeset/graph-spec-slices.md @@ -0,0 +1,5 @@ +--- +"@executor-js/plugin-openapi": patch +--- + +Serve Microsoft Graph preset selections from precomputed slice release assets instead of the 43MB upstream monolith. The monolith fetch almost never survives a 128MB Workers isolate (production traces show one completion in 30 days), so covered selections — every catalog preset, plus any combination within the default bundle — now read a 4–19MB filtered document built offline by the graph-slices workflow, with the monolith path kept only as a fallback and for full-graph/custom-scope selections. diff --git a/.changeset/lazy-mcp-client-module.md b/.changeset/lazy-mcp-client-module.md new file mode 100644 index 000000000..309303275 --- /dev/null +++ b/.changeset/lazy-mcp-client-module.md @@ -0,0 +1,5 @@ +--- +"@executor-js/plugin-mcp": patch +--- + +Load the MCP client SDK lazily on first outbound connection instead of at module evaluation. Runtimes that bundle the plugin (notably Cloudflare Workers) no longer pay the client package's module-eval memory and CPU on startup or on code paths that never dial an MCP server. diff --git a/.changeset/local-native-elicitation-streaming.md b/.changeset/local-native-elicitation-streaming.md deleted file mode 100644 index 9d5faf060..000000000 --- a/.changeset/local-native-elicitation-streaming.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -"executor": patch ---- - -**Fix: native MCP elicitation now reaches clients on the local HTTP endpoint instead of timing out** - -The local daemon's Streamable HTTP transport ran with `enableJsonResponse: true`, which buffers a `tools/call` into a single JSON body and leaves no open stream for the server to write on. A server-to-client `elicitation/create` raised during that call was therefore never delivered, and approval-gated tools failed with a `-32001` request timeout even though the session had negotiated `elicitation_mode=native` and the client's `elicitation.form` capability. The transport now uses the spec-default SSE streaming, so the reverse request rides the originating tool call's stream — matching the Cloudflare host's behaviour. diff --git a/.changeset/mcp-namespace-search-tools.md b/.changeset/mcp-namespace-search-tools.md new file mode 100644 index 000000000..28818f61b --- /dev/null +++ b/.changeset/mcp-namespace-search-tools.md @@ -0,0 +1,8 @@ +--- +"@executor-js/execution": patch +"executor": patch +--- + +**Opt-in per-integration search tools on the MCP surface** + +Connecting with `?search_tools=true` (stdio: `executor mcp --search-tools`) adds one minimally-described `search_` MCP tool per connected integration, so the integration namespaces reach the model as tool names it can see without calling anything. Each call routes through the same flow as `tools.search({ namespace })` inside `execute`, and the tool list comes from the same inventory the `execute` description shows. Off by default; a clean endpoint URL is unchanged. diff --git a/.changeset/openapi-streaming-preview.md b/.changeset/openapi-streaming-preview.md new file mode 100644 index 000000000..fb7d142fb --- /dev/null +++ b/.changeset/openapi-streaming-preview.md @@ -0,0 +1,5 @@ +--- +"@executor-js/plugin-openapi": patch +--- + +Preview OpenAPI spec-format selections (Microsoft Graph) through the streaming structural-split path instead of a whole-document parse, and guard generic whole-document parses by parsed-tree size (line count for block YAML, text size for JSON). Previewing a Graph preset URL previously parsed the 43MB source whole and killed the 128MB Workers isolate mid-request, surfacing as an empty 503; it now streams within budget, and oversized generic specs fail with an actionable error instead of taking down the isolate. diff --git a/.changeset/slim-search-tools.md b/.changeset/slim-search-tools.md new file mode 100644 index 000000000..05d45ea03 --- /dev/null +++ b/.changeset/slim-search-tools.md @@ -0,0 +1,6 @@ +--- +"@executor-js/execution": patch +"executor": patch +--- + +Slim the per-integration `search_` tool definitions to under half their size: one shared one-line description (the tool name already carries the namespace) and a single bare `query` parameter, dropping the `limit`/`offset` knobs. A session pays for these definitions once per connected integration, so the surface now costs ~2k tokens instead of ~5k at 30 integrations; paging through a namespace belongs in `execute`. diff --git a/.changeset/storage-error-shaping.md b/.changeset/storage-error-shaping.md new file mode 100644 index 000000000..7a50ec85b --- /dev/null +++ b/.changeset/storage-error-shaping.md @@ -0,0 +1,8 @@ +--- +"@executor-js/sdk": patch +"@executor-js/api": patch +--- + +Build `StorageError.message` from the call-site label plus the driver's error code instead of the driver's raw text. The driver text is drizzle's `Failed query: \nparams: `, so error reporting grouped one storage defect by statement shape and printed bound parameters into issue titles. The full driver error stays on `cause`. + +Add `StorageConnectionError`, a `StorageFailure` variant for postgres.js connection faults (`CONNECTION_ENDED`, `CONNECTION_CLOSED`, `CONNECTION_DESTROYED`, `CONNECT_TIMEOUT`, `ECONNREFUSED`, `ECONNRESET`) and workerd's cross-request I/O rejection. It carries the fault `code` and a `retryable` flag so a lost socket can be told apart from a pool-lifetime bug. diff --git a/.claude/skills/prod-telemetry/SKILL.md b/.claude/skills/prod-telemetry/SKILL.md index 47f79bbc7..0128c9ecd 100644 --- a/.claude/skills/prod-telemetry/SKILL.md +++ b/.claude/skills/prod-telemetry/SKILL.md @@ -29,6 +29,20 @@ join the same traces via traceparent). **Span names worth querying** (and their custom attrs): +- `mcp.execute` / `mcp.execute.resume` — `mcp.execute.mode` + (`pausable`/`inline`), `mcp.execute.code_length`, and + `mcp.execute.outcome` (`ok`/`fail`/`paused`) with, on failures, + `mcp.execute.error_kind` (`type_error` | `reference_error` | + `syntax_error` | `range_error` | `tool_error` | `timeout` | + `resource_limit` | `serialization_error` | `thrown` | `unknown`). + Sandbox script failures ride the MCP success channel, so `status.code` + stays OK — filter on these attributes, not span status. Spans from + before the attributes shipped carry neither; absence is not success. + Also `mcp.execute.result_chars` (compact-JSON size of the returned + value, pre-truncation; -1 = unmeasurable), `mcp.execute.log_chars`, + `mcp.execute.emitted` — the dump-vs-narrow signal (the model preview + truncates at 30k chars, so `result_chars > 30000` means the model tried + to pull a truncated blob into context). - `executor.tool.execute` — `mcp.tool.name` (full address), and since PR #992: `executor.tool.outcome` (`ok`/`fail`), `executor.tool.error_code`, `executor.tool.error_status`, @@ -39,7 +53,10 @@ join the same traces via traceparent). `base_url`, and since PR #992 `http.status_code`. - `mcp.request` (outer) — `mcp.auth.organization_id`, `mcp.auth.account_id`, `mcp.tool.name`, CF edge fields (`cf.country`…), - MCP client fingerprint (`mcp.client.name`…). + MCP client fingerprint (`mcp.client.name`…), and on managed-cloud + `execute`/`execute-action` calls `mcp.execute.code` (the script itself, + capped at 10k chars — cloud-only content capture; local/self-host + telemetry never records content). **Recipe — error signatures by class (the daily-digest query):** diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9eb2ca585..a9daf5378 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,6 +10,18 @@ concurrency: group: ci-${{ github.ref }} cancel-in-progress: true +# CI is not a user. Without this, executor's own product telemetry reports to the +# production PostHog project from every e2e shard: the analytics layer +# (integration_added, execution_completed, artifact_*) and the +# integrations-registry fetch, which lands as a `hit` event. Each shard is a +# fresh container, so it mints a fresh anonymous id and arrives on a fresh +# runner IP — inflating machine and user counts by roughly an order of +# magnitude. Both opt-outs are the packages' documented CI/test hooks. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: changes: name: Changed paths @@ -18,6 +30,7 @@ jobs: outputs: desktop_smoke: ${{ github.event_name != 'pull_request' || steps.filter.outputs.desktop_smoke == 'true' }} selfhost_docker_smoke: ${{ github.event_name != 'pull_request' || steps.filter.outputs.selfhost_docker_smoke == 'true' }} + cloud_closure: ${{ github.event_name != 'pull_request' || steps.filter.outputs.cloud_closure == 'true' }} steps: - uses: actions/checkout@v4 @@ -41,6 +54,14 @@ jobs: - "packages/kernel/runtime-quickjs/**" - "packages/plugins/**" - "packages/react/**" + cloud_closure: + - ".github/workflows/**" + - "bun.lock" + - "package.json" + - "turbo.json" + - "apps/cloud/**" + - "packages/**" + selfhost_docker_smoke: - ".github/workflows/**" - ".dockerignore" @@ -144,15 +165,18 @@ jobs: test: name: Test - runs-on: blacksmith-4vcpu-ubuntu-2404 + runs-on: blacksmith-16vcpu-ubuntu-2404 timeout-minutes: 15 - # Tuned for Blacksmith's 4 vCPU runners. + # Run eight independent packages at once and cap each Vitest process at two + # workers. This uses all 16 vCPUs without every nested runner seeing the + # whole machine and oversubscribing it. env: TURBO_API: ${{ vars.TURBO_API }} TURBO_TEAM: ${{ vars.TURBO_TEAM }} TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }} TURBO_REMOTE_CACHE_SIGNATURE_KEY: ${{ secrets.TURBO_REMOTE_CACHE_SIGNATURE_KEY }} - TURBO_TEST_CONCURRENCY: 4 + TURBO_TEST_CONCURRENCY: 8 + VITEST_MAX_WORKERS: 2 steps: - uses: actions/checkout@v4 @@ -184,6 +208,8 @@ jobs: # no signal. This is just a few file reads plus greps, so ~zero cost. - run: bun run check:patches + - run: bun run --cwd e2e test:ci-shard + - run: bun run test e2e: @@ -192,14 +218,41 @@ jobs: fail-fast: false matrix: include: - # Each cloud shard boots its own fresh dev stack. On 4 vCPU runners, - # four fatter shards keep the longest shard below selfhost while saving - # four runner boots and four warm cache restores. - - { target: cloud, shard: 1/4, shard-name: 1of4 } - - { target: cloud, shard: 2/4, shard-name: 2of4 } - - { target: cloud, shard: 3/4, shard-name: 3of4 } - - { target: cloud, shard: 4/4, shard-name: 4of4 } - - target: selfhost + # The planner assigns every file exactly once using recorded slow-file + # durations plus a conservative weight for new tests. The cloud DB's + # connection teardown and concurrent socket protocol have dedicated + # regression tests; these shards balance wall clock, not hide retries. + - { target: cloud, shard-index: 1, shard-name: 1of16 } + - { target: cloud, shard-index: 2, shard-name: 2of16 } + - { target: cloud, shard-index: 3, shard-name: 3of16 } + - { target: cloud, shard-index: 4, shard-name: 4of16 } + - { target: cloud, shard-index: 5, shard-name: 5of16 } + - { target: cloud, shard-index: 6, shard-name: 6of16 } + - { target: cloud, shard-index: 7, shard-name: 7of16 } + - { target: cloud, shard-index: 8, shard-name: 8of16 } + - { target: cloud, shard-index: 9, shard-name: 9of16 } + - { target: cloud, shard-index: 10, shard-name: 10of16 } + - { target: cloud, shard-index: 11, shard-name: 11of16 } + - { target: cloud, shard-index: 12, shard-name: 12of16 } + - { target: cloud, shard-index: 13, shard-name: 13of16 } + - { target: cloud, shard-index: 14, shard-name: 14of16 } + - { target: cloud, shard-index: 15, shard-name: 15of16 } + - { target: cloud, shard-index: 16, shard-name: 16of16 } + - { target: selfhost, shard-index: 1, shard-name: 1of10 } + - { target: selfhost, shard-index: 2, shard-name: 2of10 } + - { target: selfhost, shard-index: 3, shard-name: 3of10 } + - { target: selfhost, shard-index: 4, shard-name: 4of10 } + - { target: selfhost, shard-index: 5, shard-name: 5of10 } + - { target: selfhost, shard-index: 6, shard-name: 6of10 } + - { target: selfhost, shard-index: 7, shard-name: 7of10 } + - { target: selfhost, shard-index: 8, shard-name: 8of10 } + - { target: selfhost, shard-index: 9, shard-name: 9of10 } + - { target: selfhost, shard-index: 10, shard-name: 10of10 } + # Local files own their server, browser and data directory. Separate + # runners preserve that isolation while removing its 69-second serial + # lane from the two-minute critical path. + - { target: local, shard-index: 1, shard-name: 1of2 } + - { target: local, shard-index: 2, shard-name: 2of2 } runs-on: blacksmith-4vcpu-ubuntu-2404 timeout-minutes: 30 steps: @@ -235,26 +288,31 @@ jobs: # Install from e2e so bunx resolves ITS pinned playwright (the version # the tests run against) rather than floating to the latest. + # Blacksmith's Ubuntu image carries the official GitHub runner system + # dependencies. Restore the pinned browser binaries without apt-updating + # every matrix machine. - name: Install Playwright Chromium - run: bunx playwright install --with-deps chromium chromium-headless-shell + run: bunx playwright install chromium chromium-headless-shell working-directory: e2e - # The globalsetup boots the target's own dev server (ports are claimed - # per checkout, so this is hermetic) and tears it down after the run. - # --retry=2: browser scenarios can still hit isolated waitFor timeouts - # (single-test waitFor timeouts, not systemic failures); a retry on the - # same booted stack clears them. - - name: Run cloud scenarios + # Each target either boots its own shared dev server or lets each file own + # its server. Ports and data paths are hermetic in both cases. + # Do not retry scenarios: retries hide flakes and multiply slow timeout + # failures. The fixtures and process lifecycle are deterministic enough + # that the first result is the result. + - name: Run cloud shard if: matrix.target == 'cloud' env: MCP_SESSION_TIMEOUT_MS: "3000" - MCP_PAUSED_SESSION_IDLE_TIMEOUT_MS: "6000" - run: bunx vitest run --project cloud --retry=2 ${{ matrix.shard && format('--shard={0}', matrix.shard) || '' }} + # Still 18x shorter than production, but long enough for a cold Vite + # resume route to compile under a fully loaded CI runner. + MCP_PAUSED_SESSION_IDLE_TIMEOUT_MS: "30000" + run: bun scripts/run-ci-shard.ts cloud ${{ matrix['shard-index'] }} working-directory: e2e - - name: Run selfhost scenarios - if: matrix.target == 'selfhost' - run: bunx vitest run --project selfhost --retry=2 + - name: Run scenarios + if: matrix.target != 'cloud' + run: bun scripts/run-ci-shard.ts ${{ matrix.target }} ${{ matrix['shard-index'] }} working-directory: e2e # Failed runs keep their trace.zip / session.mp4 / step screenshots in @@ -267,13 +325,23 @@ jobs: path: e2e/runs/ retention-days: 7 - e2e-local: - name: E2E (stdio MCP) - # Skipped on pull_request: the local scenario boots a real `executor web` - # plus a browser and is currently flaky on PRs. Still runs on push to main. - if: github.event_name != 'pull_request' + cloud-closure: + name: Cloud evaluated closure + needs: changes + if: needs.changes.outputs.cloud_closure == 'true' runs-on: blacksmith-4vcpu-ubuntu-2404 timeout-minutes: 20 + env: + TURBO_API: ${{ vars.TURBO_API }} + TURBO_TEAM: ${{ vars.TURBO_TEAM }} + TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }} + TURBO_REMOTE_CACHE_SIGNATURE_KEY: ${{ secrets.TURBO_REMOTE_CACHE_SIGNATURE_KEY }} + # Every cold isolate evaluates this closure before it can answer a + # request, and it only ever grows by accident - a barrel export or a new + # module-scope import quietly pulls megabytes into the server graph. The + # budget is a ratchet just above today's size, not a discovered limit: + # when it trips, make the new dependency lazy rather than raising it. + START_CLOSURE_BUDGET_MB: 13.5 steps: - uses: actions/checkout@v4 @@ -289,41 +357,17 @@ jobs: restore-keys: | ${{ runner.os }}-bun-1.3.11- - # The local scenarios boot a real `executor web` (which spawns a Node - # sidecar) and some drive a browser, so pin Node 24 and install Chromium. - uses: actions/setup-node@v4 with: node-version: 24 - run: bun install --frozen-lockfile - - name: Cache Playwright browsers - uses: actions/cache@v4 - with: - path: ~/.cache/ms-playwright - key: ${{ runner.os }}-playwright-1.60.0 - restore-keys: | - ${{ runner.os }}-playwright- - - # `chromium` and the new `chromium-headless-shell` ship as separate - # downloads; the browser-driven scenarios launch the headless shell. - # Install from e2e so bunx resolves ITS pinned playwright (the version the - # tests run against) rather than floating to the latest, which would fetch - # a browser build the test runtime does not look for. - - name: Install Playwright Chromium - run: bunx playwright install --with-deps chromium chromium-headless-shell - working-directory: e2e + - run: bun run build + working-directory: apps/cloud - # The `local` project is excluded from the default `test` chain (each - # scenario boots its own `executor web`). Run just the stdio MCP scenario - # here: it is the auto-connect / env-as-secret regression guard, and - # running it alone avoids the boot-resource accumulation and the - # pre-existing browser flakiness of the rest of the local suite. Expanding - # to the full `local` project (bun run test:local) is a follow-up once - # those are stabilized. - - name: Run the stdio MCP scenario - run: bunx vitest run --project local local/stdio-mcp.test.ts - working-directory: e2e + - run: node scripts/start-closure.mjs + working-directory: apps/cloud desktop-smoke: name: Desktop smoke build @@ -375,7 +419,13 @@ jobs: - name: Run sidecar outside the workspace working-directory: apps/desktop run: | - docker run --rm -d --name sidecar-smoke -p 45841:45841 -e HOME=/tmp \ + # Containers do not inherit the job environment. Keep the compiled + # product smoke from becoming a production analytics/integrations hit. + docker run --rm -d --name sidecar-smoke -p 45841:45841 \ + -e HOME=/tmp \ + -e DO_NOT_TRACK=1 \ + -e EXECUTOR_DISABLE_ANALYTICS=1 \ + -e EXECUTOR_DISABLE_INTEGRATIONS_FETCH=1 \ -v "$PWD/resources/executor:/opt/executor:ro" \ debian:bookworm-slim /opt/executor/executor daemon run --foreground \ --port 45841 --hostname 0.0.0.0 --auth-token=ci-smoke diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 80bff301a..2f0d97c24 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -9,6 +9,16 @@ concurrency: group: deploy-production cancel-in-progress: false +# CI is not a user — see the note in ci.yml. Applied to every workflow rather +# than the ones that look like they run product code: the first pass guessed, +# missed preview/deploy/pkg-pr-new, and kept leaking. These vars are inert +# where the product is not executed, so the blanket application is the cheap +# structural answer. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: migrate: name: Migrate database @@ -77,12 +87,30 @@ jobs: VITE_PUBLIC_SENTRY_DSN: ${{ secrets.VITE_PUBLIC_SENTRY_DSN }} - name: Deploy cloud - run: bun run wrangler deploy -c dist/server/wrangler.json + run: bun run wrangler deploy -c dist/server/wrangler.json --var GIT_COMMIT_SHA:${{ github.sha }} working-directory: apps/cloud env: CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} + # Deploy marker: one event per deploy into the same Axiom dataset the + # worker traces land in, so a latency step-change lines up with its + # deploy in one query. Skipped (not failed) when the secret is absent, + # and never blocks the deploy. + - name: Record deploy marker in Axiom + env: + AXIOM_INGEST_TOKEN: ${{ secrets.AXIOM_INGEST_TOKEN }} + run: | + if [ -z "$AXIOM_INGEST_TOKEN" ]; then + echo "AXIOM_INGEST_TOKEN not configured; skipping deploy marker" + exit 0 + fi + curl -sf -X POST "https://api.axiom.co/v1/datasets/executor-cloud/ingest" \ + -H "Authorization: Bearer $AXIOM_INGEST_TOKEN" \ + -H "Content-Type: application/json" \ + -d "[{\"event\":\"deploy\",\"service\":\"executor-cloud\",\"commit_sha\":\"${{ github.sha }}\",\"actor\":\"${{ github.actor }}\",\"run_id\":\"${{ github.run_id }}\"}]" \ + || echo "deploy marker ingest failed (non-blocking)" + deploy-marketing: name: Deploy marketing runs-on: blacksmith-4vcpu-ubuntu-2404 diff --git a/.github/workflows/graph-slices.yml b/.github/workflows/graph-slices.yml new file mode 100644 index 000000000..35768a0ac --- /dev/null +++ b/.github/workflows/graph-slices.yml @@ -0,0 +1,47 @@ +# Refresh the Microsoft Graph slice release assets. +# +# The Graph OpenAPI monolith (~43MB) cannot be processed inside a Workers +# isolate, so the runtime reads per-selection slices published on the +# `graph-slices` release tag (see packages/plugins/openapi/src/providers/ +# microsoft/slices.ts). This workflow rebuilds the slices from the current +# upstream spec on a schedule and on demand. +name: Graph slices + +on: + schedule: + # Weekly; Microsoft's msgraph-metadata automation lands upstream refreshes + # on a similar cadence. A failed run leaves the previous assets serving. + - cron: "17 6 * * 1" + workflow_dispatch: {} + +permissions: + contents: write + +jobs: + slices: + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@v4 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: 1.3.11 + + - name: Install dependencies + run: bun install --frozen-lockfile + + - name: Generate slices + working-directory: packages/plugins/openapi + run: bun scripts/generate-graph-slices.ts --out "$RUNNER_TEMP/graph-slices" + + - name: Publish to the graph-slices release + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release view graph-slices --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1 || \ + gh release create graph-slices --repo "$GITHUB_REPOSITORY" \ + --title "Microsoft Graph slices" --latest=false \ + --notes "Generated per-preset Microsoft Graph OpenAPI slices. Data release consumed by the openapi plugin's Microsoft adapter; refreshed by the graph-slices workflow." + gh release upload graph-slices "$RUNNER_TEMP/graph-slices"/* \ + --repo "$GITHUB_REPOSITORY" --clobber diff --git a/.github/workflows/pkg-pr-new.yml b/.github/workflows/pkg-pr-new.yml index a4720cf51..9af29d804 100644 --- a/.github/workflows/pkg-pr-new.yml +++ b/.github/workflows/pkg-pr-new.yml @@ -10,6 +10,16 @@ concurrency: group: pkg-pr-new-${{ github.event.pull_request.number }} cancel-in-progress: true +# CI is not a user — see the note in ci.yml. Applied to every workflow rather +# than the ones that look like they run product code: the first pass guessed, +# missed preview/deploy/pkg-pr-new, and kept leaking. These vars are inert +# where the product is not executed, so the blanket application is the cheap +# structural answer. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: # Per-platform matrix: build the executor binary, tar it, upload to R2. # The wrapper npm package is built later by the `publish` job which just diff --git a/.github/workflows/preview-sweep.yml b/.github/workflows/preview-sweep.yml index 243e654ce..fd7ab6483 100644 --- a/.github/workflows/preview-sweep.yml +++ b/.github/workflows/preview-sweep.yml @@ -16,6 +16,16 @@ permissions: contents: read pull-requests: read +# CI is not a user — see the note in ci.yml. Applied to every workflow rather +# than the ones that look like they run product code: the first pass guessed, +# missed preview/deploy/pkg-pr-new, and kept leaking. These vars are inert +# where the product is not executed, so the blanket application is the cheap +# structural answer. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: sweep: name: Destroy previews for closed PRs diff --git a/.github/workflows/preview.yml b/.github/workflows/preview.yml index 288e33cbf..2409fd4f2 100644 --- a/.github/workflows/preview.yml +++ b/.github/workflows/preview.yml @@ -23,6 +23,16 @@ concurrency: group: preview-${{ github.event.pull_request.number }} cancel-in-progress: ${{ github.event.action != 'closed' }} +# CI is not a user — see the note in ci.yml. Applied to every workflow rather +# than the ones that look like they run product code: the first pass guessed, +# missed preview/deploy/pkg-pr-new, and kept leaking. These vars are inert +# where the product is not executed, so the blanket application is the cheap +# structural answer. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: deploy: name: Deploy preview diff --git a/.github/workflows/publish-desktop.yml b/.github/workflows/publish-desktop.yml index 090e0d6d4..54f22c7ce 100644 --- a/.github/workflows/publish-desktop.yml +++ b/.github/workflows/publish-desktop.yml @@ -21,6 +21,18 @@ concurrency: group: publish-desktop-${{ github.ref }} cancel-in-progress: false +# CI is not a user. Without this, executor's own product telemetry reports to the +# production PostHog project from every e2e shard: the analytics layer +# (integration_added, execution_completed, artifact_*) and the +# integrations-registry fetch, which lands as a `hit` event. Each shard is a +# fresh container, so it mints a fresh anonymous id and arrives on a fresh +# runner IP — inflating machine and user counts by roughly an order of +# magnitude. Both opt-outs are the packages' documented CI/test hooks. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: build: permissions: diff --git a/.github/workflows/publish-executor-package.yml b/.github/workflows/publish-executor-package.yml index 2ef813a67..fefdf8d89 100644 --- a/.github/workflows/publish-executor-package.yml +++ b/.github/workflows/publish-executor-package.yml @@ -21,6 +21,16 @@ concurrency: group: publish-executor-package-${{ github.ref }} cancel-in-progress: false +# CI is not a user — see the note in ci.yml. Applied to every workflow rather +# than the ones that look like they run product code: the first pass guessed, +# missed preview/deploy/pkg-pr-new, and kept leaking. These vars are inert +# where the product is not executed, so the blanket application is the cheap +# structural answer. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: publish: runs-on: ubuntu-latest diff --git a/.github/workflows/publish-selfhost-docker.yml b/.github/workflows/publish-selfhost-docker.yml index 51715ff86..a70192b13 100644 --- a/.github/workflows/publish-selfhost-docker.yml +++ b/.github/workflows/publish-selfhost-docker.yml @@ -16,6 +16,18 @@ concurrency: group: publish-selfhost-docker-${{ inputs.tag }} cancel-in-progress: false +# CI is not a user. Without this, executor's own product telemetry reports to the +# production PostHog project from every e2e shard: the analytics layer +# (integration_added, execution_completed, artifact_*) and the +# integrations-registry fetch, which lands as a `hit` event. Each shard is a +# fresh container, so it mints a fresh anonymous id and arrives on a fresh +# runner IP — inflating machine and user counts by roughly an order of +# magnitude. Both opt-outs are the packages' documented CI/test hooks. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: metadata: runs-on: blacksmith-4vcpu-ubuntu-2404 @@ -281,9 +293,6 @@ jobs: runs-on: blacksmith-4vcpu-ubuntu-2404 permissions: packages: write - env: - HAS_LEGACY_TOKEN: ${{ secrets.GHCR_LEGACY_TOKEN != '' }} - steps: - name: Download image digests uses: actions/download-artifact@v4 @@ -330,10 +339,28 @@ jobs: docker buildx imagetools create "${tag_args[@]}" "${sources[@]}" + # Mirrors each release to the pre-org-move namespace + # (ghcr.io/rhyssullivan/executor-selfhost), which GHCR cannot redirect. + # Deliberately a separate job: the canonical publish above never depends on + # it, and a failure here (an expired GHCR_LEGACY_TOKEN, historically) shows + # as one red job pointing at exactly what to fix. + mirror-legacy: + needs: + - metadata + - merge + runs-on: blacksmith-4vcpu-ubuntu-2404 + env: + HAS_LEGACY_TOKEN: ${{ secrets.GHCR_LEGACY_TOKEN != '' }} + + steps: - name: Skip legacy GHCR mirror (GHCR_LEGACY_TOKEN not configured) if: env.HAS_LEGACY_TOKEN != 'true' run: echo "GHCR_LEGACY_TOKEN is not configured; skipping legacy GHCR mirror." + - name: Set up Docker Buildx + if: env.HAS_LEGACY_TOKEN == 'true' + uses: docker/setup-buildx-action@v3 + - name: Log in to legacy GHCR namespace if: env.HAS_LEGACY_TOKEN == 'true' uses: docker/login-action@v3 @@ -342,7 +369,7 @@ jobs: username: rhyssullivan password: ${{ secrets.GHCR_LEGACY_TOKEN }} - - name: Mirror self-host image to legacy GHCR namespace + - name: Mirror release tags to the legacy namespace if: env.HAS_LEGACY_TOKEN == 'true' shell: bash env: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 241d4757f..171b42a14 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -13,6 +13,18 @@ concurrency: group: release-${{ github.ref }} cancel-in-progress: false +# CI is not a user. Without this, executor's own product telemetry reports to the +# production PostHog project from every e2e shard: the analytics layer +# (integration_added, execution_completed, artifact_*) and the +# integrations-registry fetch, which lands as a `hit` event. Each shard is a +# fresh container, so it mints a fresh anonymous id and arrives on a fresh +# runner IP — inflating machine and user counts by roughly an order of +# magnitude. Both opt-outs are the packages' documented CI/test hooks. +env: + DO_NOT_TRACK: "1" + EXECUTOR_DISABLE_ANALYTICS: "1" + EXECUTOR_DISABLE_INTEGRATIONS_FETCH: "1" + jobs: release: runs-on: ubuntu-latest diff --git a/.oxlintrc.jsonc b/.oxlintrc.jsonc index da06c5351..c86ee9dc5 100644 --- a/.oxlintrc.jsonc +++ b/.oxlintrc.jsonc @@ -64,6 +64,7 @@ "apps/desktop/src/main.ts", "scripts/**/*.{ts,js}", "apps/*/scripts/**/*.{ts,js}", + "packages/*/*/scripts/**/*.{ts,js}", "packages/kernel/runtime-*/src/**/*.{ts,tsx,js,mjs}", ], "rules": { diff --git a/README.md b/README.md index e66ea111d..cbf860f1b 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ and per-tool policies, then use that one catalog from any MCP-compatible agent. [Website](https://executor.sh) · [Documentation](https://executor.sh/docs) · [Discord](https://discord.gg/eF29HBHwM6) -[https://github.com/user-attachments/assets/11225f83-e848-42ba-99b2-a993bcc88dad](https://github.com/user-attachments/assets/11225f83-e848-42ba-99b2-a993bcc88dad) +![Executor demo](https://raw.githubusercontent.com/UsefulSoftwareCo/executor/main/assets/executor-demo.gif) ## Why Executor diff --git a/apps/cli/CHANGELOG.md b/apps/cli/CHANGELOG.md index 376411677..246471d27 100644 --- a/apps/cli/CHANGELOG.md +++ b/apps/cli/CHANGELOG.md @@ -1,5 +1,55 @@ # executor +## 1.6.0 + +### Patch Changes + +- [#1737](https://github.com/UsefulSoftwareCo/executor/pull/1737) [`9296f36`](https://github.com/UsefulSoftwareCo/executor/commit/9296f36a8adbfdeec700ce33c37987127857b2fd) Thanks [@RhysSullivan](https://github.com/RhysSullivan)! - Scope the `skills` tool to Executor's own documentation. Its description, argument, index, and unknown-name error now state that it serves a fixed catalog of how-to docs for this server's tools, so an agent on a host without a skill tool of its own no longer reads it as a general reader for the harness's or the user's skills. + +- Updated dependencies [[`a2d1417`](https://github.com/UsefulSoftwareCo/executor/commit/a2d141758e478274813c8c24d354e1fd0f66af49)]: + - @executor-js/sdk@1.6.0 + - @executor-js/local@1.6.0 + - @executor-js/api@1.4.63 + - @executor-js/runtime-quickjs@1.6.0 + +## 1.5.42 + +### Patch Changes + +- Updated dependencies [[`d3f0617`](https://github.com/UsefulSoftwareCo/executor/commit/d3f0617deec06c57e0d6e1479fe668f79daf977d)]: + - @executor-js/sdk@1.5.42 + - @executor-js/local@1.5.42 + - @executor-js/api@1.4.62 + - @executor-js/runtime-quickjs@1.5.42 + +## 1.5.41 + +### Patch Changes + +- [#1600](https://github.com/UsefulSoftwareCo/executor/pull/1600) [`1b5f931`](https://github.com/UsefulSoftwareCo/executor/commit/1b5f931d90b52fa9eca7b6f53359a117d757c7c1) Thanks [@RhysSullivan](https://github.com/RhysSullivan)! - **Add `integrations.remove` to the core tools so an agent can drop a catalog integration** + + `integrations.list` advertises `canRemove` per integration, but nothing on the agent surface could act on it: removal existed only on the HTTP API and the web console, so an agent that could add an integration could never take one back out. Cleaning up a catalog meant clicking through the UI once per integration. + + The core-tools plugin now contributes `integrations.remove`, taking the `slug` reported by `integrations.list` and cascading to every connection under the integration and the tools those produced. It is approval-gated, being strictly more destructive than `connections.remove`. The `removed` flag is honest rather than always-true: `false` means no catalog row matched, so an already-absent slug and a built-in namespace like `executor` are distinguishable from a real removal, and an integration pinned with `canRemove: false` is refused with `IntegrationRemovalNotAllowedError` instead of silently surviving. + +- [#1556](https://github.com/UsefulSoftwareCo/executor/pull/1556) [`f674fb8`](https://github.com/UsefulSoftwareCo/executor/commit/f674fb80eebd597f922edd5ec21b8035ab195a78) Thanks [@RhysSullivan](https://github.com/RhysSullivan)! - **Fix: native MCP elicitation now reaches clients on the local HTTP endpoint instead of timing out** + + The local daemon's Streamable HTTP transport ran with `enableJsonResponse: true`, which buffers a `tools/call` into a single JSON body and leaves no open stream for the server to write on. A server-to-client `elicitation/create` raised during that call was therefore never delivered, and approval-gated tools failed with a `-32001` request timeout even though the session had negotiated `elicitation_mode=native` and the client's `elicitation.form` capability. The transport now uses the spec-default SSE streaming, so the reverse request rides the originating tool call's stream — matching the Cloudflare host's behaviour. + +- [#1603](https://github.com/UsefulSoftwareCo/executor/pull/1603) [`624e85f`](https://github.com/UsefulSoftwareCo/executor/commit/624e85f033632a7624c2bddf0944112166b1f481) Thanks [@RhysSullivan](https://github.com/RhysSullivan)! - **Fix: `oauth.clients.remove` reported success for clients it never removed** + + The tool returned `{ removed: true }` unconditionally. `oauth.removeClient` is idempotent by design at the storage layer — `deleteMany` on a missing row is a no-op, which is the right behaviour for a delete — but the tool mapped that silence to success, so a typo'd slug, an already-deleted client, and the wrong owner were all indistinguishable from a real deletion. + + This bites hardest because clients are keyed by BOTH owner and slug, so the same slug can exist separately under `org` and `user`. An agent sweeping a list of slugs under one hardcoded owner would delete only half of them and report every call as a success, leaving org-owned OAuth apps registered after everything they authorized was gone. + + The tool now checks the caller-visible client set first and returns `removed: false` when nothing matched that `(owner, slug)` pair. The service-level `removeClient` is unchanged and stays idempotent. + +- Updated dependencies [[`d572658`](https://github.com/UsefulSoftwareCo/executor/commit/d572658d74097917412256f10a3ea2e3974f44dd)]: + - @executor-js/sdk@1.5.41 + - @executor-js/local@1.5.41 + - @executor-js/api@1.4.61 + - @executor-js/runtime-quickjs@1.5.41 + ## 1.5.40 ### Patch Changes diff --git a/apps/cli/package.json b/apps/cli/package.json index 572da191f..a21ce368d 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -1,6 +1,6 @@ { "name": "executor", - "version": "1.5.40", + "version": "1.6.0", "private": true, "bin": { "executor": "./bin/executor.ts" diff --git a/apps/cli/src/main.ts b/apps/cli/src/main.ts index 47d3227ca..fc2a9bf39 100644 --- a/apps/cli/src/main.ts +++ b/apps/cli/src/main.ts @@ -215,9 +215,11 @@ const waitForShutdownSignal = () => const shutdown = () => resume(Effect.void); process.once("SIGINT", shutdown); process.once("SIGTERM", shutdown); + process.once("SIGHUP", shutdown); return Effect.sync(() => { process.off("SIGINT", shutdown); process.off("SIGTERM", shutdown); + process.off("SIGHUP", shutdown); }); }); @@ -1360,6 +1362,7 @@ const mcpUrlForActiveLocalServer = (input: { readonly connection: ExecutorServerConnection; readonly elicitationMode: "browser" | "model"; readonly artifacts: boolean; + readonly searchTools: boolean; }): URL => { const url = new URL("/mcp", input.connection.origin); if (input.elicitationMode === "browser") { @@ -1370,6 +1373,11 @@ const mcpUrlForActiveLocalServer = (input: { if (!input.artifacts) { url.searchParams.set("artifacts", "false"); } + // Per-integration search tools are off by default; only the opt-in is + // spelled out. + if (input.searchTools) { + url.searchParams.set("search_tools", "true"); + } return url; }; @@ -1385,6 +1393,7 @@ const runMcpHttpBridge = async (input: { readonly manifest: ExecutorLocalServerManifest; readonly elicitationMode: "browser" | "model"; readonly artifacts: boolean; + readonly searchTools: boolean; }): Promise => { const stdio = new StdioServerTransport(); const authorization = getExecutorServerAuthorizationHeader(input.manifest.connection); @@ -1393,6 +1402,7 @@ const runMcpHttpBridge = async (input: { connection: input.manifest.connection, elicitationMode: input.elicitationMode, artifacts: input.artifacts, + searchTools: input.searchTools, }), authorization ? { requestInit: { headers: { Authorization: authorization } } } : undefined, ); @@ -1471,6 +1481,7 @@ const runMcpHttpBridge = async (input: { const runStdioMcpSession = (input: { readonly elicitationMode: "browser" | "model"; readonly artifacts: boolean; + readonly searchTools: boolean; }) => Effect.gen(function* () { // `executor mcp` never owns the local database. If a local server is already @@ -1487,6 +1498,7 @@ const runStdioMcpSession = (input: { manifest: active, elicitationMode: input.elicitationMode, artifacts: input.artifacts, + searchTools: input.searchTools, }), ); return; @@ -1513,6 +1525,7 @@ const runStdioMcpSession = (input: { manifest: elected, elicitationMode: input.elicitationMode, artifacts: input.artifacts, + searchTools: input.searchTools, }), ); }); @@ -2878,11 +2891,18 @@ const mcpCommand = Command.make( "Withhold the artifact surface from this connection: the artifact tools, the app shell resource, and the artifact skills. Served by default.", ), ), + searchTools: Options.boolean("search-tools") + .pipe(Options.withDefault(false)) + .pipe( + Options.withDescription( + "Serve one search_ tool per connected integration. Off by default; each routes through the same flow as tools.search inside execute.", + ), + ), }, - ({ scope, elicitationMode, noArtifacts }) => + ({ scope, elicitationMode, noArtifacts, searchTools }) => Effect.gen(function* () { applyScope(scope); - yield* runStdioMcpSession({ elicitationMode, artifacts: !noArtifacts }); + yield* runStdioMcpSession({ elicitationMode, artifacts: !noArtifacts, searchTools }); }), ).pipe(Command.withDescription("Start an MCP server over stdio")); diff --git a/apps/cloud/CHANGELOG.md b/apps/cloud/CHANGELOG.md index acabecc71..69496f303 100644 --- a/apps/cloud/CHANGELOG.md +++ b/apps/cloud/CHANGELOG.md @@ -1,5 +1,68 @@ # @executor-js/cloud +## 1.4.61 + +### Patch Changes + +- Updated dependencies [[`c11bef2`](https://github.com/UsefulSoftwareCo/executor/commit/c11bef2cd049db7bbf51b15e18761b14acccb534), [`46cea2c`](https://github.com/UsefulSoftwareCo/executor/commit/46cea2cbb1f414ae58ac876819a51b11967909a6), [`a2d1417`](https://github.com/UsefulSoftwareCo/executor/commit/a2d141758e478274813c8c24d354e1fd0f66af49), [`2bdbedf`](https://github.com/UsefulSoftwareCo/executor/commit/2bdbedf257f54d7c209e8c856c618174c10d6bb3), [`0b0b74f`](https://github.com/UsefulSoftwareCo/executor/commit/0b0b74f673b8098c5248159be36c648097f3c87b), [`256e25e`](https://github.com/UsefulSoftwareCo/executor/commit/256e25e7b291b0c023bc7547d092004b66781bba)]: + - @executor-js/plugin-mcp@1.6.0 + - @executor-js/plugin-openapi@1.6.0 + - @executor-js/sdk@1.6.0 + - @executor-js/react@1.4.63 + - @executor-js/runtime-dynamic-worker@1.4.4 + - @executor-js/api@1.4.63 + - @executor-js/execution@1.6.0 + - @executor-js/vite-plugin@0.0.60 + - @executor-js/cloudflare@0.0.42 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.11 + - @executor-js/plugin-graphql@1.6.0 + - @executor-js/plugin-toolkits@1.5.35 + - @executor-js/plugin-workos-vault@0.0.2 + - @executor-js/runtime-quickjs@1.6.0 + +## 1.4.60 + +### Patch Changes + +- Updated dependencies [[`d3f0617`](https://github.com/UsefulSoftwareCo/executor/commit/d3f0617deec06c57e0d6e1479fe668f79daf977d), [`86c68af`](https://github.com/UsefulSoftwareCo/executor/commit/86c68afef9bf8b7c19ab58f59acfedca0b3c4ca7), [`32206c7`](https://github.com/UsefulSoftwareCo/executor/commit/32206c7f78654f638bfd27c25c71c30c3d6354be), [`9ecc7cb`](https://github.com/UsefulSoftwareCo/executor/commit/9ecc7cb8b30375ffa960e3fefe4d211e0254e691)]: + - @executor-js/sdk@1.5.42 + - @executor-js/cloudflare@0.0.41 + - @executor-js/plugin-openapi@1.5.42 + - @executor-js/plugin-mcp@1.5.42 + - @executor-js/api@1.4.62 + - @executor-js/execution@1.5.42 + - @executor-js/vite-plugin@0.0.59 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.10 + - @executor-js/runtime-dynamic-worker@1.4.4 + - @executor-js/plugin-graphql@1.5.42 + - @executor-js/plugin-toolkits@1.5.34 + - @executor-js/plugin-workos-vault@0.0.2 + - @executor-js/react@1.4.62 + - @executor-js/runtime-quickjs@1.5.42 + +## 1.4.59 + +### Patch Changes + +- Updated dependencies [[`d572658`](https://github.com/UsefulSoftwareCo/executor/commit/d572658d74097917412256f10a3ea2e3974f44dd), [`a9b33d2`](https://github.com/UsefulSoftwareCo/executor/commit/a9b33d25c32fbb4a292b7e8963e22392f862a16f)]: + - @executor-js/sdk@1.5.41 + - @executor-js/plugin-mcp@1.5.41 + - @executor-js/api@1.4.61 + - @executor-js/execution@1.5.41 + - @executor-js/vite-plugin@0.0.58 + - @executor-js/cloudflare@0.0.40 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.9 + - @executor-js/runtime-dynamic-worker@1.4.4 + - @executor-js/plugin-graphql@1.5.41 + - @executor-js/plugin-openapi@1.5.41 + - @executor-js/plugin-toolkits@1.5.33 + - @executor-js/plugin-workos-vault@0.0.2 + - @executor-js/react@1.4.61 + - @executor-js/runtime-quickjs@1.5.41 + ## 1.4.58 ### Patch Changes diff --git a/apps/cloud/package.json b/apps/cloud/package.json index 6ce693125..a9add7028 100644 --- a/apps/cloud/package.json +++ b/apps/cloud/package.json @@ -1,6 +1,6 @@ { "name": "@executor-js/cloud", - "version": "1.4.58", + "version": "1.4.61", "private": true, "type": "module", "scripts": { diff --git a/apps/cloud/scripts/dev-db.ts b/apps/cloud/scripts/dev-db.ts index 482c86900..ec671a4d9 100644 --- a/apps/cloud/scripts/dev-db.ts +++ b/apps/cloud/scripts/dev-db.ts @@ -15,6 +15,7 @@ import { PGlite } from "@electric-sql/pglite"; import { PGLiteSocketServer } from "@electric-sql/pglite-socket"; import { drizzle } from "drizzle-orm/pglite"; import { migrate } from "drizzle-orm/pglite/migrator"; +import postgres from "postgres"; const __dirname = dirname(fileURLToPath(import.meta.url)); // Port + data dir default to the dev values but are env-overridable so a second @@ -104,25 +105,46 @@ await migrate(drizzle(db), { migrationsFolder: MIGRATIONS_FOLDER }); // but prepared statement requires M" -> random 500s on whichever request lost // the race). The patch in patches/@electric-sql%2Fpglite-socket@0.1.4.patch // batches each socket data event into one queue entry and holds handler -// affinity while a pipeline is open; -// src/db/dev-db-socket-concurrency.node.test.ts is the regression test. -const server = new PGLiteSocketServer({ - db, - port: PORT, - host: "127.0.0.1", - maxConnections: Number(process.env.DEV_DB_MAX_CONNECTIONS ?? 1000), - // Backstop for pipeline affinity: a client that stalls mid-pipeline (Parse - // sent, no Sync) with its socket still OPEN would hold the queue's handler - // affinity forever and starve every other connection, since affinity only - // releases on detach and detach needs close/error/idle-timeout. In ms; the - // timer resets on every data event, so only a genuinely dead client trips it. - idleTimeout: Number(process.env.DEV_DB_IDLE_TIMEOUT_MS ?? 30_000), -}); +// affinity while a pipeline is open. The patch also fixes the queue's two +// self-bricking failure paths — both surfaced in CI as the e2e "cloud signIn: +// callback set no session (500)" cascade, where new connections' startup +// packets sat unanswered (postgres.js CONNECT_TIMEOUT) until restart: +// 1. Stock 0.1.4 `return`ed out of the drain loop when a query REJECTED at +// the JS level, leaving its `processing` flag latched true; nothing was +// ever dequeued again. +// 2. A client whose socket died WHILE its pipeline-opening entry executed: +// detach() cleared affinity before the entry finished, the queue then +// took affinity for the already-dead handler, and no timer was left to +// release it. The queue now tracks detached handlers and repairs any +// transaction or pipeline affinity they can no longer release. +// src/db/dev-db-socket-concurrency.node.test.ts is the regression test for +// all of the above. +const makeServer = () => + new PGLiteSocketServer({ + db, + port: PORT, + host: "127.0.0.1", + maxConnections: Number(process.env.DEV_DB_MAX_CONNECTIONS ?? 1000), + // Backstop for pipeline affinity: a client that stalls mid-pipeline (Parse + // sent, no Sync) with its socket still OPEN would hold the queue's handler + // affinity forever and starve every other connection, since affinity only + // releases on detach and detach needs close/error/idle-timeout. In ms; the + // timer resets on every data event. The patch scopes the reap to connections + // actually HOLDING affinity (open pipeline or transaction): an idle-at-rest + // connection is the normal state of a healthy postgres.js pool held by a + // long-lived scope (SSE), and reaping those raced live queries into + // sporadic `write CONNECTION_ENDED` 500s. + idleTimeout: Number(process.env.DEV_DB_IDLE_TIMEOUT_MS ?? 30_000), + }); +let server = makeServer(); await server.start(); console.log(`[dev-db] Listening on postgresql://postgres:postgres@127.0.0.1:${PORT}/postgres`); +let stopping = false; + const shutdown = async () => { + stopping = true; console.log("\n[dev-db] Shutting down"); await server.stop(); await db.close(); @@ -131,3 +153,112 @@ const shutdown = async () => { process.on("SIGINT", shutdown); process.on("SIGTERM", shutdown); + +// --------------------------------------------------------------------------- +// Wedge watchdog +// --------------------------------------------------------------------------- +// +// Twice the socket server has shipped a state machine that could stop +// answering NEW connections while the process, the port, and PGlite all stayed +// up (the CI e2e "cloud signIn: callback set no session (500)" cascades: every +// in-flight query dies once, then every fresh connection's startup packet +// times out — CONNECT_TIMEOUT — for the rest of the shard). The known paths +// are patched with regression tests, but each recurrence so far has found a +// new path, and a wedged front-end turns ONE infra hiccup into a failure of +// every remaining test in the shard. +// +// So: probe the server the way the app does — a fresh TCP connection, real +// startup handshake, `select 1` — and when several consecutive probes fail, +// dump the server's internals to the boot log and swap in a fresh socket +// server on the same PGlite instance (all state is in PGlite; the front-end is +// stateless, so this drops only already-doomed connections). If a restart +// doesn't restore service, exit non-zero: the boot supervisor logs the exit +// loudly and the run fails fast with an attributable cause instead of minutes +// of anonymous CONNECT_TIMEOUTs. The wedge itself stays visible in the +// server-logs artifact via the [dev-db][watchdog] lines. +const WATCHDOG_INTERVAL_MS = Number(process.env.DEV_DB_WATCHDOG_INTERVAL_MS ?? 5_000); +// 3 consecutive failures ≈ 15s+ of hard unavailability. PGlite serves queries +// in milliseconds; even a deep queue clears in well under one probe interval, +// so consecutive startup failures this sustained only happen wedged. +const WATCHDOG_FAILURES_TO_RESTART = 3; +const WATCHDOG_MAX_RESTARTS = 3; + +const probe = async (): Promise => { + const sql = postgres(`postgres://postgres:postgres@127.0.0.1:${PORT}/postgres`, { + max: 1, + idle_timeout: 0, + connect_timeout: 5, + fetch_types: false, + prepare: false, + onnotice: () => undefined, + }); + try { + // connect_timeout only bounds the handshake; race the query too so a + // post-startup wedge cannot hang the watchdog itself. + await Promise.race([ + sql.unsafe("select 1"), + sleep(10_000).then(() => { + throw new Error("probe query timed out after 10s"); + }), + ]); + } finally { + await sql.end({ timeout: 5 }).catch(() => {}); + } +}; + +const watchdog = async () => { + let consecutiveFailures = 0; + let restarts = 0; + for (;;) { + await sleep(WATCHDOG_INTERVAL_MS); + if (stopping) return; + try { + await probe(); + consecutiveFailures = 0; + } catch (cause) { + consecutiveFailures += 1; + console.error( + `[dev-db][watchdog] probe failed (${consecutiveFailures}/${WATCHDOG_FAILURES_TO_RESTART}): ${String(cause)}`, + ); + if (consecutiveFailures < WATCHDOG_FAILURES_TO_RESTART) continue; + console.error( + `[dev-db][watchdog] socket server wedged; stats: ${JSON.stringify(server.getStats())}`, + ); + if (restarts >= WATCHDOG_MAX_RESTARTS) { + console.error( + `[dev-db][watchdog] still wedged after ${restarts} restarts — giving up so the boot supervisor reports it`, + ); + process.exit(1); + } + restarts += 1; + consecutiveFailures = 0; + console.error( + `[dev-db][watchdog] restarting socket server (${restarts}/${WATCHDOG_MAX_RESTARTS})`, + ); + // stop() itself goes through the query queue (detach rolls back open + // transactions), so a wedge deep enough can hang the restart too — + // bound it and treat that as fatal rather than hanging the watchdog. + const restart = async () => { + await server.stop(); + server = makeServer(); + await server.start(); + }; + try { + await Promise.race([ + restart(), + sleep(15_000).then(() => { + throw new Error("restart timed out after 15s"); + }), + ]); + console.error(`[dev-db][watchdog] socket server restarted`); + } catch (restartCause) { + console.error( + `[dev-db][watchdog] restart failed (${String(restartCause)}) — exiting so the boot supervisor reports it`, + ); + process.exit(1); + } + } + } +}; + +void watchdog(); diff --git a/apps/cloud/scripts/start-closure.mjs b/apps/cloud/scripts/start-closure.mjs new file mode 100644 index 000000000..a841cb597 --- /dev/null +++ b/apps/cloud/scripts/start-closure.mjs @@ -0,0 +1,124 @@ +// --------------------------------------------------------------------------- +// Measure the Worker's *evaluated* module closures from the build output. +// --------------------------------------------------------------------------- +// +// Upload size does not predict cold-isolate cost: a Worker can ship megabytes +// that never load, and the bytes that matter are the ones an isolate must +// evaluate before it can answer. Rollup already separates static from dynamic +// edges, so we can compute that split directly instead of reading totals. +// +// Two closures matter: +// startup - statically reachable from the Worker entry. Evaluated on every +// cold isolate before any request is served. +// start - the TanStack Start server graph, reached through the lazy +// `loadEntries` dynamic imports. Evaluated on the first request +// that enters the app - i.e. a page request. +// app - the Effect app plane. `/api/*` dispatches at the Worker entry and +// skips Start entirely, so an API request evaluates this instead of +// `start`; reported separately because the two planes now diverge. +// +// Anything reachable only through a dynamic import is not counted: making a +// heavy dependency lazy is exactly the outcome this rewards. +// +// Scope note: this measures bytes, which is a proxy for cold-start cost, not a +// proven cause of any particular regression. The Aug 2026 MCP incident was +// NOT explained by this number - reverting the offending packages moved the +// evaluated closure by 0.02 MB while restoring production latency, so +// module-scope execution cost, not size, drove that one. Treat a budget breach +// as "this will make cold starts worse", not as "this is why the site is slow". +import { readFileSync, readdirSync, statSync } from "node:fs"; +import { dirname, join, relative, resolve } from "node:path"; + +const DIST = resolve(process.argv[2] ?? "dist/server"); +const ENTRY = join(DIST, "index.js"); + +// Rollup emits `from "./x.js"`, bare `import "./x.js"`, `export ... from "./x.js"` +// (all static) and `import("./x.js")` (dynamic). Matching the emitted output +// rather than source means we see the graph as the runtime sees it. +const STATIC_RE = /(?:from|import)\s*["'](\.[^"']+)["']/g; +const DYNAMIC_RE = /import\(\s*["'](\.[^"']+)["']\s*\)/g; + +const listChunks = (dir) => + readdirSync(dir, { withFileTypes: true }).flatMap((e) => + e.isDirectory() + ? listChunks(join(dir, e.name)) + : e.name.endsWith(".js") + ? [join(dir, e.name)] + : [], + ); + +const graph = new Map(); +for (const file of listChunks(DIST)) { + const code = readFileSync(file, "utf8"); + const dynamic = new Set([...code.matchAll(DYNAMIC_RE)].map((m) => resolve(dirname(file), m[1]))); + // A specifier inside `import(...)` also matches STATIC_RE's `import` branch, + // so subtract the dynamic set rather than trusting the static matches alone. + const staticDeps = new Set( + [...code.matchAll(STATIC_RE)] + .map((m) => resolve(dirname(file), m[1])) + .filter((p) => !dynamic.has(p)), + ); + graph.set(file, { size: statSync(file).size, static: staticDeps, dynamic }); +} + +/** Bytes evaluated when `roots` are loaded, following static edges only. */ +const closure = (roots) => { + const seen = new Set(); + const queue = [...roots]; + while (queue.length > 0) { + const file = queue.pop(); + if (seen.has(file) || !graph.has(file)) continue; + seen.add(file); + queue.push(...graph.get(file).static); + } + return seen; +}; + +const bytes = (files) => [...files].reduce((sum, f) => sum + (graph.get(f)?.size ?? 0), 0); +const mb = (n) => `${(n / 1024 / 1024).toFixed(2)} MB`; +const name = (f) => relative(DIST, f); + +const startup = closure([ENTRY]); +// The lazy server-graph entries Start pulls on first request. +const startRoots = [...graph.get(ENTRY).dynamic].filter((f) => + /(start|router|tanstack)/.test(name(f)), +); +const start = closure(startRoots); +const appRoots = [...graph.get(ENTRY).dynamic].filter((f) => /\/app-[A-Za-z0-9_-]+\.js$/.test(f)); +const app = closure([ENTRY, ...appRoots]); +// The budget tracks the worst plane: whichever costs a cold isolate more. +const evaluated = new Set([...startup, ...start]); + +const report = (label, files) => { + console.log(`\n${label}: ${mb(bytes(files))} (${files.size} chunks)`); + const own = [...files].filter((f) => !startup.has(f) || label === "startup"); + for (const f of own.sort((a, b) => graph.get(b).size - graph.get(a).size).slice(0, 12)) { + console.log(` ${(graph.get(f).size / 1024).toFixed(0).padStart(6)} KB ${name(f)}`); + } +}; + +report("startup", startup); +report("start", start); +console.log(`\npage request (startup + start): ${mb(bytes(evaluated))}`); +console.log( + `API request (startup + app): ${mb(bytes(app))}${appRoots.length ? "" : " [no app chunk - /api still routes through Start]"}`, +); +const lazyOnly = [...graph.keys()].filter((f) => !evaluated.has(f)); +console.log( + `deferred behind dynamic import: ${mb(bytes(lazyOnly))} (${lazyOnly.length} chunks)`, +); + +const budget = Number(process.env.START_CLOSURE_BUDGET_MB ?? 0); +if (budget > 0) { + const actual = bytes(evaluated) / 1024 / 1024; + console.log(`\nbudget ${budget} MB — actual ${actual.toFixed(2)} MB`); + if (actual > budget) { + console.error( + `\nFAIL: evaluated closure ${actual.toFixed(2)} MB exceeds the ${budget} MB budget.\n` + + `Every cold isolate pays to evaluate this closure before it can answer. Move the\n` + + `new weight behind a dynamic import rather than raising the budget; run this\n` + + `script with no budget set to see the biggest members and what is already lazy.`, + ); + process.exit(1); + } +} diff --git a/apps/cloud/scripts/test-globalsetup.ts b/apps/cloud/scripts/test-globalsetup.ts index d7dd916ed..4362c3285 100644 --- a/apps/cloud/scripts/test-globalsetup.ts +++ b/apps/cloud/scripts/test-globalsetup.ts @@ -13,12 +13,26 @@ import { fileURLToPath } from "node:url"; const __dirname = dirname(fileURLToPath(import.meta.url)); -const PORT = 5434; +const parsePort = (input: string | undefined): number => { + if (input === undefined) return 5434; + if (!/^\d+$/.test(input)) throw new Error("CLOUD_TEST_DB_PORT must be an integer"); + const port = Number(input); + if (!Number.isSafeInteger(port) || port < 1 || port > 65_535) { + throw new Error("CLOUD_TEST_DB_PORT must be between 1 and 65535"); + } + return port; +}; + +const PORT = parsePort(process.env.CLOUD_TEST_DB_PORT); const MIGRATIONS_FOLDER = resolve(__dirname, "../drizzle"); let db: PGlite | undefined; let server: PGLiteSocketServer | undefined; +/** + * Starts the cloud unit-test database and returns teardown that releases every + * resource without allowing PGlite shutdown to erase Vitest's failure status. + */ export default async function setup() { db = await PGlite.create(); await migrate(drizzle(db), { migrationsFolder: MIGRATIONS_FOLDER }); @@ -30,7 +44,12 @@ export default async function setup() { console.log(`[test-db] PGlite socket server listening on 127.0.0.1:${PORT}`); return async () => { + // PGlite sets an internal 99 sentinel on startup and replaces it with 0 on + // close. Preserve Vitest's failure status across that close so + // teardown can turn neither a red test green nor a green test red. + const testsFailed = process.exitCode === 1; await server?.stop(); await db?.close(); + if (testsFailed) process.exitCode = 1; }; } diff --git a/apps/cloud/src/account/org-api-key-revoke.node.test.ts b/apps/cloud/src/account/org-api-key-revoke.node.test.ts index 5deb62f6b..0de1c64d2 100644 --- a/apps/cloud/src/account/org-api-key-revoke.node.test.ts +++ b/apps/cloud/src/account/org-api-key-revoke.node.test.ts @@ -6,6 +6,7 @@ import { AccountError, AccountForbidden } from "@executor-js/api"; import { ApiKeyService, OrgApiKeyNotFound } from "../auth/api-keys"; import { UserStoreService } from "../auth/context"; +import { ORG_SELECTOR_HEADER } from "../auth/organization"; import { WorkOSClient, type WorkOSClientService } from "../auth/workos"; import { AutumnService } from "../extensions/billing/service"; import { AccountCaller, workosAccountProvider } from "./workos-account-service"; @@ -36,6 +37,7 @@ const MEMBER = "user_member"; const ORG_KEY = "key_org_1"; const USER_KEY = "key_user_1"; const createdAt = new Date("2026-01-01T00:00:00.000Z"); +const orgHeaders = { [ORG_SELECTOR_HEADER]: ORG }; const session = (accountId: string) => ({ accountId, @@ -77,7 +79,7 @@ const stubWorkOS = Layer.succeed( ); const stubUsers = Layer.succeed(UserStoreService)({ - use: (fn) => + use: (_op, fn) => Effect.promise(() => fn({ ensureAccount: async (id: string) => ({ id, createdAt }), @@ -106,6 +108,7 @@ const stubUsers = Layer.succeed(UserStoreService)({ const stubAutumn = Layer.succeed(AutumnService)({ use: () => Effect.die("revoke does not touch billing"), + ensureCustomer: () => Effect.die("revoke does not touch billing"), checkExecutionBalance: () => Effect.die("revoke does not touch billing"), trackExecution: () => Effect.void, }); @@ -157,7 +160,7 @@ describe("revokeOrgApiKey · provider boundary", () => { const { provider, revoked } = providerWith(ADMIN); const account = yield* provider; - const result = yield* account.revokeOrgApiKey({}, ORG_KEY); + const result = yield* account.revokeOrgApiKey(orgHeaders, ORG_KEY); expect(result).toEqual({ success: true }); expect(revoked, "the revoke reached the key service").toEqual([ORG_KEY]); @@ -169,7 +172,7 @@ describe("revokeOrgApiKey · provider boundary", () => { const { provider, revoked } = providerWith(MEMBER); const account = yield* provider; - const error = yield* Effect.flip(account.revokeOrgApiKey({}, ORG_KEY)); + const error = yield* Effect.flip(account.revokeOrgApiKey(orgHeaders, ORG_KEY)); expect(error, "same admin gate as the mint").toBeInstanceOf(AccountForbidden); expect(revoked, "the gate runs BEFORE the key service is touched").toEqual([]); @@ -183,7 +186,7 @@ describe("revokeOrgApiKey · provider boundary", () => { const { provider, revoked } = providerWith(ADMIN); const account = yield* provider; - const error = yield* Effect.flip(account.revokeOrgApiKey({}, USER_KEY)); + const error = yield* Effect.flip(account.revokeOrgApiKey(orgHeaders, USER_KEY)); expect(error).toBeInstanceOf(AccountError); expect(revoked).toEqual([]); diff --git a/apps/cloud/src/account/workos-account-service.ts b/apps/cloud/src/account/workos-account-service.ts index 739e91d27..6e70a5b9a 100644 --- a/apps/cloud/src/account/workos-account-service.ts +++ b/apps/cloud/src/account/workos-account-service.ts @@ -386,7 +386,9 @@ export const workosAccountProvider: Layer.Layer< .updateOrganization(org.id, name) .pipe(Effect.catchTag("WorkOSError", toAccountError)); yield* users - .use((s) => s.upsertOrganization({ id: updated.id, name: updated.name })) + .use("upsertOrganization", (s) => + s.upsertOrganization({ id: updated.id, name: updated.name }), + ) .pipe(Effect.catchTag("UserStoreError", toAccountError)); return { name: updated.name }; }), diff --git a/apps/cloud/src/analytics/ema-rollout.test.ts b/apps/cloud/src/analytics/ema-rollout.test.ts new file mode 100644 index 000000000..c62321b05 --- /dev/null +++ b/apps/cloud/src/analytics/ema-rollout.test.ts @@ -0,0 +1,394 @@ +// --------------------------------------------------------------------------- +// Cloud's PostHog-backed enterprise-managed rollout gate. +// +// The gate is exercised through its production seam: the real +// `makePostHogEnterpriseManagedRollout` with an injected `fetch` and an +// injected `waitUntil`, exactly as `auth/jwks-cache.node.test.ts` drives the +// JWKS client. Nothing is module-mocked, so the request these tests read is the +// request PostHog would receive. +// --------------------------------------------------------------------------- + +import { describe, expect, it } from "@effect/vitest"; +import { Effect } from "effect"; + +import type { EnterpriseManagedRolloutContext } from "@executor-js/sdk"; + +import { + ENTERPRISE_MANAGED_AUTH_FLAG_KEY, + cloudEnterpriseManagedRollout, + makePostHogEnterpriseManagedRollout, +} from "./ema-rollout"; + +const HOST = "https://us.i.posthog.com"; +const PROJECT_KEY = "phc_test_project_key"; + +const CONTEXT: EnterpriseManagedRolloutContext = { + userId: "user_01ABC", + organizationId: "org_01XYZ", + integration: "slack" as EnterpriseManagedRolloutContext["integration"], +}; + +interface CapturedRequest { + readonly url: string; + readonly body: Record; + /** The bound the gate put on this call, when it set one. */ + readonly signal: AbortSignal | null; +} + +interface FetchHarness { + readonly fetch: typeof globalThis.fetch; + readonly requests: () => readonly CapturedRequest[]; + /** Resolve every request the gate detached, so an assertion cannot race it. */ + readonly settle: () => Promise; + readonly waitUntil: (work: Promise) => void; +} + +/** + * Stands in for PostHog. `respond` decides what each call returns; throwing + * from it models a transport failure, and an aborted signal models the timeout. + */ +const makeFetchHarness = ( + respond: (request: CapturedRequest) => Response | Promise, +): FetchHarness => { + const requests: CapturedRequest[] = []; + const detached: Promise[] = []; + return { + requests: () => requests, + settle: async () => { + await Promise.all(detached); + }, + waitUntil: (work) => { + detached.push(work); + }, + fetch: async (input, init) => { + const raw = typeof init?.body === "string" ? init.body : "{}"; + const captured: CapturedRequest = { + url: String(input), + // oxlint-disable-next-line executor/no-json-parse -- boundary: test fixture reads back the exact JSON body the gate serialized for PostHog + body: JSON.parse(raw) as Record, + signal: init?.signal instanceof AbortSignal ? init.signal : null, + }; + requests.push(captured); + return respond(captured); + }, + }; +}; + +const jsonResponse = (body: unknown, status = 200): Response => + new Response(JSON.stringify(body), { + status, + headers: { "content-type": "application/json" }, + }); + +const flagsBody = (enabled: boolean) => ({ + flags: { + [ENTERPRISE_MANAGED_AUTH_FLAG_KEY]: { + key: ENTERPRISE_MANAGED_AUTH_FLAG_KEY, + enabled, + reason: { code: "condition_match" }, + }, + }, + errorsWhileComputingFlags: false, +}); + +const gate = (harness: FetchHarness, timeoutMs?: number) => + makePostHogEnterpriseManagedRollout({ + projectKey: PROJECT_KEY, + host: HOST, + fetch: harness.fetch, + waitUntil: harness.waitUntil, + ...(timeoutMs === undefined ? {} : { timeoutMs }), + }); + +describe("flag evaluation", () => { + it.effect("keys the rollout on the user and carries the org as group context", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse(flagsBody(true))); + + const decision = yield* gate(harness).decide(CONTEXT); + + expect(decision).toEqual({ kind: "enabled" }); + const [request] = harness.requests(); + expect(request?.url).toBe(`${HOST}/flags?v=2`); + expect( + request?.body.distinct_id, + "the rollout unit is the user, matching posthog.identify(user.id) in the browser", + ).toBe(CONTEXT.userId); + expect( + request?.body.groups, + "the org rides along so group targeting stays available without changing the rollout unit", + ).toEqual({ organization: CONTEXT.organizationId }); + expect(request?.body.api_key).toBe(PROJECT_KEY); + expect(request?.signal, "the evaluation is always bounded").not.toBeNull(); + }), + ); + + it.effect("omits group context when the host named no organization", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse(flagsBody(true))); + + yield* gate(harness).decide({ ...CONTEXT, organizationId: null }); + + expect(harness.requests()[0]?.body.groups).toBeUndefined(); + }), + ); + + it.effect("withholds when the flag is off for this user", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse(flagsBody(false))); + + expect(yield* gate(harness).decide(CONTEXT)).toEqual({ + kind: "withheld", + reason: "disabled", + }); + }), + ); + + it.effect("withholds when PostHog returned no verdict for the flag at all", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => + // What a quota-limited project answers. + jsonResponse({ + flags: {}, + errorsWhileComputingFlags: false, + quotaLimited: ["feature_flags"], + }), + ); + + expect(yield* gate(harness).decide(CONTEXT)).toEqual({ + kind: "withheld", + reason: "disabled", + }); + }), + ); +}); + +describe("failing closed", () => { + it.effect("withholds on a non-2xx answer", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => new Response("nope", { status: 503 })); + + expect(yield* gate(harness).decide(CONTEXT)).toEqual({ + kind: "withheld", + reason: "evaluation-unavailable", + }); + }), + ); + + it.effect("withholds on a transport failure", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => { + // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- boundary: models `fetch` rejecting with a TypeError, which is exactly how a Worker sees a dead upstream + throw new TypeError("network error"); + }); + + expect(yield* gate(harness).decide(CONTEXT)).toEqual({ + kind: "withheld", + reason: "evaluation-unavailable", + }); + }), + ); + + it.effect("withholds when the evaluation times out", () => + Effect.gen(function* () { + // Never answers. Only the gate's own AbortSignal ends this call, so the + // test fails by hanging if the gate ever stops bounding the evaluation. + const harness = makeFetchHarness( + (request) => + new Promise((_resolve, reject) => { + request.signal?.addEventListener("abort", () => + // oxlint-disable-next-line executor/no-promise-reject, executor/no-error-constructor -- boundary: fetch-compatible fixture mirrors the platform's abort rejection semantics + reject(new DOMException("aborted", "AbortError")), + ); + }), + ); + + expect( + yield* gate(harness, 1).decide(CONTEXT), + "a slow flag service must degrade the rollout, never hold up a connect", + ).toEqual({ kind: "withheld", reason: "evaluation-unavailable" }); + }), + ); + + it.effect("withholds on a body that is not a flags response", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse({ flags: "not-an-object" })); + + expect(yield* gate(harness).decide(CONTEXT)).toEqual({ + kind: "withheld", + reason: "evaluation-unavailable", + }); + }), + ); + + it.effect("withholds when there is no acting user to roll out by", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse(flagsBody(true))); + + expect(yield* gate(harness).decide({ ...CONTEXT, userId: null })).toEqual({ + kind: "withheld", + reason: "evaluation-unavailable", + }); + expect(harness.requests().length, "and asks PostHog nothing").toBe(0); + }), + ); + + it.effect("withholds when the deployment has no PostHog configuration", () => + Effect.gen(function* () { + // The cloud test env carries no VITE_PUBLIC_POSTHOG_KEY, so this builds + // the misconfigured-deployment gate — which is deliberately NOT the same + // as a host that injects no gate at all. + expect(yield* cloudEnterpriseManagedRollout().decide(CONTEXT)).toEqual({ + kind: "withheld", + reason: "evaluation-unavailable", + }); + }), + ); +}); + +describe("rollout events", () => { + it.effect("captures the connect outcome with the flag decision attached", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse({ status: 1 })); + const rollout = gate(harness); + + yield* rollout.record({ + kind: "connected", + context: CONTEXT, + decision: { kind: "enabled" }, + }); + yield* Effect.promise(() => harness.settle()); + + const [request] = harness.requests(); + expect(request?.url).toBe(`${HOST}/i/v0/e/`); + expect(request?.body.event).toBe("ema_connect_connected"); + expect(request?.body.distinct_id).toBe(CONTEXT.userId); + const properties = request?.body.properties as Record; + expect(properties.ema_flag_enabled).toBe(true); + expect(properties[`$feature/${ENTERPRISE_MANAGED_AUTH_FLAG_KEY}`]).toBe(true); + expect(properties.$groups).toEqual({ + organization: CONTEXT.organizationId, + }); + expect(properties.integration_slug).toBe("slack"); + }), + ); + + it.effect("carries the withheld reason on the attempt that never ran", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse({ status: 1 })); + + yield* gate(harness).record({ + kind: "attempted", + context: CONTEXT, + decision: { kind: "withheld", reason: "evaluation-unavailable" }, + }); + yield* Effect.promise(() => harness.settle()); + + const properties = harness.requests()[0]?.body.properties as Record; + expect(harness.requests()[0]?.body.event).toBe("ema_connect_attempted"); + expect(properties.ema_flag_enabled).toBe(false); + expect(properties.ema_flag_withheld_reason).toBe("evaluation-unavailable"); + }), + ); + + it.effect("carries the identity provider's own error code on a blocked connect", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse({ status: 1 })); + + yield* gate(harness).record({ + kind: "blocked-by-admin", + context: CONTEXT, + decision: { kind: "enabled" }, + oauthErrorCode: "unauthorized_client", + }); + yield* Effect.promise(() => harness.settle()); + + const [request] = harness.requests(); + expect(request?.body.event).toBe("ema_connect_blocked_by_admin"); + const properties = request?.body.properties as Record | undefined; + expect(properties?.oauth_error_code).toBe("unauthorized_client"); + }), + ); + + it.effect("sends nothing beyond identity, the integration and the flag decision", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse({ status: 1 })); + + yield* gate(harness).record({ + kind: "blocked-by-admin", + context: CONTEXT, + decision: { kind: "enabled" }, + oauthErrorCode: "access_denied", + }); + yield* Effect.promise(() => harness.settle()); + + const properties = harness.requests()[0]?.body.properties as Record; + expect( + Object.keys(properties).sort(), + "a closed property set is what keeps a token or an assertion from ever being added by accident", + ).toEqual( + [ + "$feature/mcp-enterprise-managed-auth", + "$groups", + "ema_flag_enabled", + "integration_slug", + "oauth_error_code", + ].sort(), + ); + }), + ); + + it.effect("hands the capture to the platform instead of awaiting it", () => + Effect.gen(function* () { + let resolveCapture: (() => void) | undefined; + const harness = makeFetchHarness( + () => + new Promise((resolve) => { + resolveCapture = () => resolve(jsonResponse({ status: 1 })); + }), + ); + + // Returns while the capture is still in flight: an analytics call can + // never be on the critical path of a connect. + yield* gate(harness).record({ + kind: "attempted", + context: CONTEXT, + decision: { kind: "enabled" }, + }); + + expect(harness.requests().length).toBe(1); + resolveCapture?.(); + yield* Effect.promise(() => harness.settle()); + }), + ); + + it.effect("survives a capture that fails outright", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => { + // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- boundary: models the ingest endpoint being unreachable + throw new TypeError("network error"); + }); + + yield* gate(harness).record({ + kind: "connected", + context: CONTEXT, + decision: { kind: "enabled" }, + }); + yield* Effect.promise(() => harness.settle()); + }), + ); + + it.effect("records nothing when there is no person to attach the event to", () => + Effect.gen(function* () { + const harness = makeFetchHarness(() => jsonResponse({ status: 1 })); + + yield* gate(harness).record({ + kind: "attempted", + context: { ...CONTEXT, userId: null }, + decision: { kind: "withheld", reason: "evaluation-unavailable" }, + }); + + expect(harness.requests().length).toBe(0); + }), + ); +}); diff --git a/apps/cloud/src/analytics/ema-rollout.ts b/apps/cloud/src/analytics/ema-rollout.ts new file mode 100644 index 000000000..88d10e8e9 --- /dev/null +++ b/apps/cloud/src/analytics/ema-rollout.ts @@ -0,0 +1,255 @@ +// --------------------------------------------------------------------------- +// Cloud's rollout gate for MCP Enterprise-Managed Authorization. +// +// Implements the vendor-free `EnterpriseManagedRollout` port from +// `@executor-js/sdk` against PostHog, which is the only flag/analytics service +// this deployment operates. Two deliberate shapes: +// +// 1. NO SDK. This is a hand-written `fetch` against two documented PostHog +// endpoints, not `posthog-node`. Cloud already ate an unexplained 3-5s +// page-load regression from adding one dependency to the worker bundle and +// the mechanism was never identified (see the notes on the MCP SDK v2 +// revert). Until that is understood, a new runtime dependency in this bundle +// is a cost we are not willing to pay for a boolean, and the whole client we +// would be importing is two POSTs wide. +// +// 2. FAIL CLOSED, ALWAYS. Every way of not getting an answer — timeout, +// transport failure, non-2xx, a body we cannot parse, no PostHog key +// configured, no acting user to key the rollout on — withholds the +// enterprise-managed path and falls back to ordinary interactive OAuth. A +// PostHog outage therefore degrades the rollout and never fails a connect, +// and (because the SDK freezes the verdict onto the connection) never +// touches an enterprise-managed connection that already exists. +// +// The rollout UNIT is the user: `distinct_id` is the acting user's id, the same +// id `posthog.identify(user.id, …)` uses in the browser, so a percentage +// rollout here means the same population it means everywhere else in the +// project. The organization rides along as group context so org-level targeting +// stays available in the PostHog UI without changing that unit. +// --------------------------------------------------------------------------- + +import { env } from "cloudflare:workers"; +import { Effect, Schema } from "effect"; + +import type { + EnterpriseManagedRollout, + EnterpriseManagedRolloutContext, + EnterpriseManagedRolloutDecision, + EnterpriseManagedRolloutEvent, +} from "@executor-js/sdk"; + +import { POSTHOG_INGEST_HOST } from "../edge/passthrough"; + +/** The flag that gates enterprise-managed authorization. */ +export const ENTERPRISE_MANAGED_AUTH_FLAG_KEY = "mcp-enterprise-managed-auth"; + +/** PostHog group type for organizations — the same one + * `posthog.group("organization", …)` establishes in the browser. */ +const ORGANIZATION_GROUP_TYPE = "organization"; + +/** + * How long a flag evaluation may take before the gate gives up and withholds. + * Short on purpose: this call sits in front of an interactive connect, and the + * safe answer is already known, so waiting is strictly worse than answering. + */ +export const DEFAULT_FLAG_EVALUATION_TIMEOUT_MS = 1_000; + +/** Wire names of the rollout events, in the product's `object_verb` style. */ +const EVENT_NAMES = { + attempted: "ema_connect_attempted", + connected: "ema_connect_connected", + "blocked-by-admin": "ema_connect_blocked_by_admin", +} as const satisfies Record; + +// --------------------------------------------------------------------------- +// Boundary parsing +// --------------------------------------------------------------------------- + +/** The slice of PostHog's `POST /flags?v=2` response this gate reads. Unknown + * keys are ignored, so PostHog adding fields cannot make the gate fail closed; + * a body that is not this shape at all can, which is the intent. */ +const FlagsResponse = Schema.Struct({ + flags: Schema.Record(Schema.String, Schema.Struct({ enabled: Schema.Boolean })), +}); + +const decodeFlagsResponse = Schema.decodeUnknownEffect(FlagsResponse); + +// --------------------------------------------------------------------------- +// Detached work +// --------------------------------------------------------------------------- + +/** + * Hand a request to the platform and stop caring about it. + * + * `waitUntil` is the correct owner when the caller has an `ExecutionContext`; + * the executor composition seam this gate is installed through does not receive + * one, so the default owner attaches a terminal rejection handler and lets the + * in-flight request ride out the rest of the response. Either way the promise + * is owned by something that cannot report back, which is what makes an event + * structurally unable to fail or delay a connect. + */ +const detach = ( + work: Promise, + waitUntil: ((work: Promise) => void) | undefined, +): void => { + const settled = work.then( + () => undefined, + () => undefined, + ); + if (waitUntil) waitUntil(settled); +}; + +// --------------------------------------------------------------------------- +// The gate +// --------------------------------------------------------------------------- + +export interface PostHogRolloutConfig { + /** Public project key (`phc_…`). Public by design — it is already shipped to + * every browser — so this is a var, not a secret. */ + readonly projectKey: string; + /** PostHog API origin (no trailing slash), e.g. `https://us.i.posthog.com`. */ + readonly host: string; + readonly fetch: typeof globalThis.fetch; + readonly timeoutMs?: number; + /** Platform post-response hook, when the caller has one. */ + readonly waitUntil?: (work: Promise) => void; +} + +const withheld = ( + reason: "disabled" | "evaluation-unavailable", +): EnterpriseManagedRolloutDecision => ({ kind: "withheld", reason }); + +const ENABLED: EnterpriseManagedRolloutDecision = { kind: "enabled" }; + +/** Group context for a connect whose organization the host named. */ +const groupsFor = (context: EnterpriseManagedRolloutContext): Record | undefined => + context.organizationId === null + ? undefined + : { [ORGANIZATION_GROUP_TYPE]: context.organizationId }; + +/** + * Build cloud's enterprise-managed rollout gate over an explicit PostHog + * configuration. Everything environmental is a parameter so the composition + * root parses it once and tests drive the real code path with an injected + * `fetch`. + */ +export const makePostHogEnterpriseManagedRollout = ( + config: PostHogRolloutConfig, +): EnterpriseManagedRollout => { + const timeoutMs = config.timeoutMs ?? DEFAULT_FLAG_EVALUATION_TIMEOUT_MS; + + const evaluate = Effect.fn("Cloud.EnterpriseManagedRollout.decide")(function* ( + context: EnterpriseManagedRolloutContext, + ) { + // The rollout unit is the user. With no acting user there is no + // `distinct_id` to key it on, so there is no answer to be had — withhold + // rather than invent a rollout bucket. + const distinctId = context.userId; + if (distinctId === null) return withheld("evaluation-unavailable"); + + const groups = groupsFor(context); + const response = yield* Effect.tryPromise(() => + config.fetch(`${config.host}/flags?v=2`, { + method: "POST", + headers: { + "content-type": "application/json", + // Declares this a server-side evaluation, which is what selects + // server-runtime flags in PostHog's runtime detection. + "user-agent": "executor-cloud", + }, + body: JSON.stringify({ + api_key: config.projectKey, + distinct_id: distinctId, + ...(groups === undefined ? {} : { groups }), + }), + signal: AbortSignal.timeout(timeoutMs), + }), + ); + if (!response.ok) return withheld("evaluation-unavailable"); + + const body = yield* Effect.tryPromise(() => response.json() as Promise); + const decoded = yield* decodeFlagsResponse(body); + const flag = decoded.flags[ENTERPRISE_MANAGED_AUTH_FLAG_KEY]; + // A flag PostHog did not return is a flag that is not on for this user + // (it may not exist yet, or the project may be quota-limited). That is a + // real "no", not a failure to answer. + if (flag === undefined || !flag.enabled) return withheld("disabled"); + return ENABLED; + }); + + return { + decide: (context) => + evaluate(context).pipe( + // Timeout, transport failure and an unparseable body all land here and + // all mean the same thing: no answer, so no enterprise-managed attempt. + // `catchCause` rather than `catchAll` because a defect in this adapter + // must not become a failed connect either. + Effect.catchCause(() => Effect.succeed(withheld("evaluation-unavailable"))), + ), + record: (event) => + Effect.sync(() => { + const distinctId = event.context.userId; + // Same reason `decide` withholds: an event with no person to attach to + // would only pollute the project with anonymous rows. + if (distinctId === null) return; + const groups = groupsFor(event.context); + const decision = event.decision; + detach( + config.fetch(`${config.host}/i/v0/e/`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + api_key: config.projectKey, + event: EVENT_NAMES[event.kind], + distinct_id: distinctId, + properties: { + // Product metadata only. There is no field on + // `EnterpriseManagedRolloutEvent` that can carry a token, an + // identity assertion, a client secret or a scope, and none is + // synthesized here. + integration_slug: String(event.context.integration), + ema_flag_enabled: decision.kind === "enabled", + ...(decision.kind === "withheld" + ? { ema_flag_withheld_reason: decision.reason } + : {}), + // PostHog's own convention, so these events can be filtered by + // flag value in the UI alongside every other flag. + [`$feature/${ENTERPRISE_MANAGED_AUTH_FLAG_KEY}`]: decision.kind === "enabled", + ...(event.kind === "blocked-by-admin" && event.oauthErrorCode !== undefined + ? { oauth_error_code: event.oauthErrorCode } + : {}), + ...(groups === undefined ? {} : { $groups: groups }), + }, + }), + }), + config.waitUntil, + ); + }), + }; +}; + +/** A gate that answers "no" to everything, for a deployment that has no PostHog + * configuration to evaluate against. Cloud is SUPPOSED to have one, so a + * missing key is a misconfiguration, and failing closed is the same choice the + * outage paths make. This is NOT the same as injecting no gate at all, which + * is how the hosts with no flag service (local, desktop, CLI, self-host) keep + * their existing behavior. */ +export const unavailableEnterpriseManagedRollout: EnterpriseManagedRollout = { + decide: () => Effect.succeed(withheld("evaluation-unavailable")), + record: () => Effect.void, +}; + +/** + * The gate as the cloud worker composes it: the public project key already + * shipped in `wrangler.jsonc`, and the same PostHog ingest origin the + * adblock-dodging passthrough proxies to. + */ +export const cloudEnterpriseManagedRollout = (): EnterpriseManagedRollout => { + const projectKey = env.VITE_PUBLIC_POSTHOG_KEY; + if (!projectKey) return unavailableEnterpriseManagedRollout; + return makePostHogEnterpriseManagedRollout({ + projectKey, + host: env.VITE_PUBLIC_POSTHOG_HOST ?? `https://${POSTHOG_INGEST_HOST}`, + fetch: (input, init) => globalThis.fetch(input, init), + }); +}; diff --git a/apps/cloud/src/api/protected-api-key-auth.node.test.ts b/apps/cloud/src/api/protected-api-key-auth.node.test.ts index 602713334..3ba01a61b 100644 --- a/apps/cloud/src/api/protected-api-key-auth.node.test.ts +++ b/apps/cloud/src/api/protected-api-key-auth.node.test.ts @@ -47,7 +47,7 @@ const stubWorkOS = Layer.succeed( ); const stubUsers = Layer.succeed(UserStoreService)({ - use: (fn) => + use: (_op, fn) => Effect.promise(() => fn({ ensureAccount: async (id: string) => ({ id, createdAt }), diff --git a/apps/cloud/src/api/protected-jwt-auth.node.test.ts b/apps/cloud/src/api/protected-jwt-auth.node.test.ts index 2120c9004..dbf35e1c2 100644 --- a/apps/cloud/src/api/protected-jwt-auth.node.test.ts +++ b/apps/cloud/src/api/protected-jwt-auth.node.test.ts @@ -64,7 +64,7 @@ const stubWorkOS = Layer.succeed( ); const stubUsers = Layer.succeed(UserStoreService)({ - use: (fn) => + use: (_op, fn) => Effect.promise(() => fn({ ensureAccount: async (id: string) => ({ id, createdAt }), diff --git a/apps/cloud/src/app-paths.test.ts b/apps/cloud/src/app-paths.test.ts index 98aeff3bd..dd9ff1af3 100644 --- a/apps/cloud/src/app-paths.test.ts +++ b/apps/cloud/src/app-paths.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "@effect/vitest"; -import { isAppOwnedPath } from "./app-paths"; +import { isAppOwnedPath, servedByAppPlane } from "./app-paths"; // Guards the start.ts dispatch decision: every surface the unified app handler // serves must be classified app-owned (forwarded to `app.handler`), and Start's @@ -62,3 +62,39 @@ describe("isAppOwnedPath", () => { }); } }); + +describe("app-plane dispatch", () => { + // These two are the whole risk of dispatching `/api` before Start: both still + // return a response if routed early, just the wrong one, so nothing else would + // catch a regression here. + it("leaves the Sentry tunnel POST to Start's middleware", () => { + expect(servedByAppPlane("/api/sentry-tunnel", "POST")).toBe(false); + // Only the POST is claimed; anything else under that path is ordinary API. + expect(servedByAppPlane("/api/sentry-tunnel", "GET")).toBe(true); + }); + + it("leaves the OAuth callback to Start, for the signed-out redirect", () => { + expect(servedByAppPlane("/api/oauth/callback", "GET")).toBe(false); + expect(servedByAppPlane("/api/oauth/callback", "POST")).toBe(false); + }); + + const appPlane = [ + "/api/connections", + "/api/tools", + "/api/integrations", + "/api/account/members", + "/api/docs", + "/api/billing/checkout", + ]; + for (const pathname of appPlane) { + it(`serves ${pathname} without entering Start`, () => { + expect(servedByAppPlane(pathname, "GET")).toBe(true); + }); + } + + it("never claims a non-API path, however app-owned", () => { + expect(servedByAppPlane("/mcp", "POST")).toBe(false); + expect(servedByAppPlane("/", "GET")).toBe(false); + expect(servedByAppPlane("/.well-known/oauth-authorization-server", "GET")).toBe(false); + }); +}); diff --git a/apps/cloud/src/app-paths.ts b/apps/cloud/src/app-paths.ts index 6dd6fc56f..48e70a8c3 100644 --- a/apps/cloud/src/app-paths.ts +++ b/apps/cloud/src/app-paths.ts @@ -17,3 +17,31 @@ export const isApiPath = (pathname: string) => pathname === "/api" || pathname.s export const isAppOwnedPath = (pathname: string) => isApiPath(pathname) || classifyMcpPath(pathname) !== null; + +// --------------------------------------------------------------------------- +// Which plane serves an app-owned path: the Effect app directly, or TanStack +// Start's middleware chain. +// +// Everything under `/api` is pure Effect and touches no part of the router, +// React, or SSR — so `server.ts` dispatches it at the Worker entry and skips +// Start's lazy `loadEntries` import entirely. Two paths must NOT take that +// shortcut, because Start's request middleware claims them BEFORE the app +// handler would ever see them: +// +// POST /api/sentry-tunnel - `sentryTunnelMiddleware` forwards the envelope +// to Sentry; the app has no such route. +// /api/oauth/callback - `oauthCallbackSignInMiddleware` redirects a +// signed-out visitor to /login, and start.ts +// rewrites the org-scoped `state` before handing +// off. Routing it early would drop both. +// +// Getting this wrong is silent: the request still gets a response, just the +// wrong one, which is why it is classified here and tested rather than being +// an inline condition at the dispatch site. +// --------------------------------------------------------------------------- + +export const isStartOwnedApiPath = (pathname: string, method: string): boolean => + (pathname === "/api/sentry-tunnel" && method === "POST") || pathname === "/api/oauth/callback"; + +export const servedByAppPlane = (pathname: string, method: string): boolean => + isApiPath(pathname) && !isStartOwnedApiPath(pathname, method); diff --git a/apps/cloud/src/app.ts b/apps/cloud/src/app.ts index 77afa9c7f..19b0213ea 100644 --- a/apps/cloud/src/app.ts +++ b/apps/cloud/src/app.ts @@ -13,7 +13,6 @@ import { ApiKeyService } from "./auth/api-keys"; import { cloudIdentityFailureStrategy, workosIdentityLayer } from "./auth/workos-auth-provider"; import { DbService } from "./db/db"; import { cloudMcpAuth } from "./mcp"; -import { McpSessionDOSqlite } from "./mcp/session-durable-object"; import { ErrorCaptureLive } from "./observability"; import { AutumnService } from "./extensions/billing/service"; import { @@ -31,7 +30,7 @@ import { WorkerTelemetryLive } from "./observability/telemetry"; // The whole scenario in 60 seconds: WorkOS identity (api-key Bearer OR sealed- // session cookie, api-key wins) over a per-request Hyperdrive→Postgres socket, // the Cloudflare dynamic-worker code substrate, MCP served by a Durable-Object -// session store (the DO surfaced via `config.mcpExport`), console+Sentry error +// session store (the DO class is exported straight from server.ts), console+Sentry error // capture — and Autumn BILLING entering ONLY as extensions: the engine // metering decorator, the account seat-gate, the `/api/billing/*` proxy route, // and the createOrganization free-limit gate. `diff` against @@ -66,7 +65,7 @@ const controlPlane = Layer.mergeAll(CoreSharedServices, apiKeyService); // satisfied by `boot`, `DbService` per request via `requestScoped`). const cloudDb: Layer.Layer = CloudDbProvider; -const { appLayer, toWebHandler, mcpExport } = ExecutorApp.make({ +const { appLayer, toWebHandler } = ExecutorApp.make({ plugins: cloudPlugins, providers: { // Identity: the NEUTRAL `IdentityProvider`. WorkOS api-key Bearer BEATS @@ -112,7 +111,6 @@ const { appLayer, toWebHandler, mcpExport } = ExecutorApp.make({ failure: cloudIdentityFailureStrategy, // The MCP session Durable Object class — a top-level Workers export a Layer // can't return; surfaced so `server.ts` can re-export it. - mcpExport: McpSessionDOSqlite, }, // The long-lived (boot-scoped) context provideMerge'd under everything: the // WorkOS control plane (the raw `WorkOSClient` + `ApiKeyService` the per-request @@ -135,10 +133,7 @@ const { appLayer, toWebHandler, mcpExport } = ExecutorApp.make({ requestScoped: RequestScopedServicesLive, }); -export { McpSessionDOSqlite }; - export const CloudAppLayer = appLayer; -export const cloudMcpExport = mcpExport; // The unified cloud web handler: serves /api/* (incl. /api/billing/*, /api/docs), // /mcp, /.well-known/* — everything the worker dispatches. diff --git a/apps/cloud/src/auth/api.ts b/apps/cloud/src/auth/api.ts index 9d14c8d64..5c64be3c5 100644 --- a/apps/cloud/src/auth/api.ts +++ b/apps/cloud/src/auth/api.ts @@ -181,6 +181,15 @@ const McpApprovalErrors = [ /** Public auth endpoints — no authentication required */ export class CloudAuthPublicApi extends HttpApiGroup.make("cloudAuthPublic") .add(HttpApiEndpoint.get("login", "/auth/login", { query: AuthLoginSearch })) + // Sign-out is PUBLIC on purpose. The console posts it as a top-level form + // navigation (the WorkOS hop is cross-origin, so it can't be a fetch), which + // means an error response is rendered as the page — and a browser whose + // session has already ended is exactly the browser most likely to click it + // (a second tab, a re-submit from history, an expired or revoked session). + // Behind SessionAuth all of those got a raw `{"_tag":"Unauthorized"}` screen + // instead of being signed out. There is nothing to authorize here anyway: + // the request can only end the session whose cookie it presents. + .add(HttpApiEndpoint.post("logout", "/auth/logout")) .add( HttpApiEndpoint.get("callback", "/auth/callback", { query: AuthCallbackSearch, @@ -201,7 +210,6 @@ export class CloudAuthApi extends HttpApiGroup.make("cloudAuth") error: AuthErrors, }), ) - .add(HttpApiEndpoint.post("logout", "/auth/logout")) .add( HttpApiEndpoint.get("organizations", "/auth/organizations", { success: AuthOrganizationsResponse, diff --git a/apps/cloud/src/auth/context.ts b/apps/cloud/src/auth/context.ts index bfd3ea25c..ce8aa1804 100644 --- a/apps/cloud/src/auth/context.ts +++ b/apps/cloud/src/auth/context.ts @@ -1,7 +1,7 @@ import { Context, Effect, Layer } from "effect"; import { makeUserStore } from "../auth/user-store"; import { DbService } from "../db/db"; -import { UserStoreError, tryPromiseService, withServiceLogging } from "./errors"; +import { tryPromiseService, userStoreErrorFromFailure, withServiceLogging } from "./errors"; // --------------------------------------------------------------------------- // UserStoreService — wraps the Drizzle-backed user store with Effect @@ -9,11 +9,16 @@ import { UserStoreError, tryPromiseService, withServiceLogging } from "./errors" type RawStore = ReturnType; +// `op` names the store call so every span reads `user_store.` +// instead of one undifferentiated "user_store" bucket, failures log which +// query actually failed, and — because the same `op` is threaded onto the +// public error alongside the classified driver reason — an error report is +// diagnosable without the trace. const makeService = (store: RawStore) => ({ - use: (fn: (s: RawStore) => Promise) => + use: (op: string, fn: (s: RawStore) => Promise) => withServiceLogging( - "user_store", - () => new UserStoreError(), + `user_store.${op}`, + (failure) => userStoreErrorFromFailure(op, failure), tryPromiseService(() => fn(store)), ), }); diff --git a/apps/cloud/src/auth/ssr-gate.ts b/apps/cloud/src/auth/doc-gate.ts similarity index 89% rename from apps/cloud/src/auth/ssr-gate.ts rename to apps/cloud/src/auth/doc-gate.ts index a95c225df..16d6145bc 100644 --- a/apps/cloud/src/auth/ssr-gate.ts +++ b/apps/cloud/src/auth/doc-gate.ts @@ -1,19 +1,23 @@ // --------------------------------------------------------------------------- -// SSR auth gate — the server-side session check for DOCUMENT requests. +// Document auth gate — the server-side session check for DOCUMENT requests. // // The sealed `wos-session` cookie is verified right here in the worker // (unseal + JWT check against cached JWKS — no per-request WorkOS round trip -// except token refresh), so by the time the SPA is served the server KNOWS -// who it's serving: +// except token refresh), so by the time the SPA shell is served the server +// KNOWS who it's serving: // // - signed out → 302 /login (carrying ?returnTo=) before any app HTML exists // - org-less → 302 /create-org (onboarding owns those sessions) -// - signed in → the document is served WITH the verified identity: the -// auth-hint travels to the SSR render via request-middleware context (the -// root loader picks it up), and is minted as a cookie when the browser -// doesn't hold a current one — so the very first paint is the real app -// shell, never a skeleton. The hint is display-only; /account/me remains -// the authority and the client keeps it fresh from then on. +// - signed in → the prerendered shell is served, and the auth-hint cookie is +// minted when the browser doesn't hold a current one — the client seeds its +// auth state from that cookie right after mount (there is no per-request +// render to dehydrate it into). The hint is display-only; /account/me +// remains the authority and the client keeps it fresh from then on. +// +// Beyond redirects, this gate is load-bearing for SESSION LIFETIME: WorkOS +// refresh tokens are single-use, and a verify that rotates the sealed session +// must deliver the new cookie to the browser or the next expiry logs the user +// out. Document navigations are where that rotation lands. // // Scope: GET/HEAD requests that are document navigations (sec-fetch-dest / // accept), excluding app-owned paths (/api, /mcp — they answer for themselves @@ -39,7 +43,6 @@ import { parseCookie } from "./cookies"; import { LAST_ORG_COOKIE } from "./last-org-cookie"; import { sealedSessionDisplayName } from "./middleware"; import { authorizeOrganizationSelector } from "./organization"; -import { browserOriginFromRequest } from "./request-origin"; import { loginPath, safeReturnTo } from "./return-to"; import { ONBOARDING_PATHS, PUBLIC_PATHS } from "./route-paths"; import { WorkOSClient } from "./workos"; @@ -153,7 +156,7 @@ const organizationDisplay = async ( ): Promise<{ name: string; slug: string }> => { const exit = await getRuntime().runPromiseExit( Effect.flatMap(UserStoreService.asEffect(), (users) => - users.use((store) => store.getOrganization(organizationId)), + users.use("getOrganization", (store) => store.getOrganization(organizationId)), ).pipe(Effect.provide(Layer.provide(makeUserStoreLayer(), makeDbLayer()))), ); return Exit.isSuccess(exit) @@ -279,19 +282,11 @@ export const authGateMiddleware = createMiddleware({ type: "request" }).server( } } - // Serve the document WITH the verified identity: the hint rides to the - // SSR render through middleware context (the root loader reads it), so - // the server paints the real authenticated shell — no loading state, no - // skeleton. The request origin rides along too: it's what the connect - // card's MCP URL is built from, and the server knows it (the SPA only - // learns `window.location.origin` after mount), so passing it here lets - // SSR render the real `https://…//mcp` instead of the client-side - // `http://127.0.0.1:4000` default — which would otherwise flash until - // hydration corrected it. Set-cookie writes ride on the rendered response. + // Serve the shell, minting the auth-hint cookie when the browser lacks a + // current one so the client's post-mount cookie read seeds the verified + // identity. Set-cookie writes ride on the shell response. const { hint, mint } = await resolveAuthHint(session, cookieHeader); - const result = await next({ - context: { authHint: hint, origin: browserOriginFromRequest(request) }, - }); + const result = await next(); if (!mint && !session.refreshedSession) return result; const response = new Response(result.response.body, result.response); diff --git a/apps/cloud/src/auth/errors.ts b/apps/cloud/src/auth/errors.ts index 927ae59f6..debf0b063 100644 --- a/apps/cloud/src/auth/errors.ts +++ b/apps/cloud/src/auth/errors.ts @@ -1,10 +1,103 @@ import { Data, Effect, Option, Predicate, Schema } from "effect"; +// How a user-store call failed, classified from the driver cause. Safe to put +// on the wire and on a Sentry tag: it names a failure MODE, never a query, a +// value, or a customer. +export const USER_STORE_FAILURE_REASONS = [ + "connect_timeout", + "connection_closed", + "query", + "unknown", +] as const; + +export type UserStoreFailureReason = (typeof USER_STORE_FAILURE_REASONS)[number]; + +/** + * The public failure of every cloud user-store call. + * + * It carries the two fields that make an issue diagnosable from the error + * alone: which store call failed, and how. Before those existed the error had + * an empty field set, so Sentry showed a titleless, messageless issue and the + * only cause detail (the pretty-printed Effect cause in a Sentry `extra`) is + * scrubbed server-side — the failing operation and the driver reason existed + * only in the trace store. Same shape as `WorkOSError.status`: a small, safe + * classification field threaded at the service boundary. + */ export class UserStoreError extends Schema.TaggedErrorClass()( "UserStoreError", - {}, + { + /** The store call that failed, e.g. `getOrganization`. */ + operation: Schema.String, + /** How it failed, classified from the driver cause chain. */ + reason: Schema.Literals(USER_STORE_FAILURE_REASONS), + }, { httpApiStatus: 500 }, -) {} +) { + override get message(): string { + return `user store ${this.operation} failed: ${this.reason}`; + } +} + +/** Reasons a retry can plausibly clear: the query never reached a healthy + * server. A `query` failure is deterministic and must not be retried. */ +export const isTransientUserStoreReason = (reason: UserStoreFailureReason): boolean => + reason === "connect_timeout" || reason === "connection_closed"; + +// postgres.js tags its connection failures with a string `code` +// (`CONNECT_TIMEOUT`, `CONNECTION_CLOSED`, …) and its query failures with the +// SQLSTATE. Drizzle re-throws both wrapped in its own "Failed query" error with +// the driver error in `.cause`, and the service adapter wraps that again, so +// the classification walks the chain rather than inspecting one level. +const MAX_CAUSE_DEPTH = 8; + +const REASON_BY_DRIVER_CODE: Readonly> = { + CONNECT_TIMEOUT: "connect_timeout", + ETIMEDOUT: "connect_timeout", + CONNECTION_CLOSED: "connection_closed", + CONNECTION_ENDED: "connection_closed", + CONNECTION_DESTROYED: "connection_closed", + ECONNREFUSED: "connection_closed", + ECONNRESET: "connection_closed", +}; + +const stringCodeOf = (value: unknown): string | undefined => { + if (typeof value !== "object" || value === null) return undefined; + const code = (value as { readonly code?: unknown }).code; + return typeof code === "string" ? code : undefined; +}; + +const driverCodesOf = (failure: unknown): readonly string[] => { + const codes: string[] = []; + let current: unknown = isServiceAdapterError(failure) ? failure.cause : failure; + for ( + let depth = 0; + depth < MAX_CAUSE_DEPTH && current !== undefined && current !== null; + depth++ + ) { + const code = stringCodeOf(current); + if (code !== undefined) codes.push(code); + current = + typeof current === "object" ? (current as { readonly cause?: unknown }).cause : undefined; + } + return codes; +}; + +/** Classify a raw store failure. A recognised connection code wins; any other + * driver code (a SQLSTATE) is a deterministic query failure; nothing at all is + * `unknown`. */ +export const userStoreReasonFromCause = (failure: unknown): UserStoreFailureReason => { + const codes = driverCodesOf(failure); + for (const code of codes) { + const reason = REASON_BY_DRIVER_CODE[code]; + if (reason !== undefined) return reason; + } + return codes.length > 0 ? "query" : "unknown"; +}; + +/** Build the public `UserStoreError` for a store-adapter failure, naming the + * operation the call site already knows and classifying the driver cause. */ +export const userStoreErrorFromFailure = (operation: string, failure: unknown): UserStoreError => + new UserStoreError({ operation, reason: userStoreReasonFromCause(failure) }); export class WorkOSError extends Schema.TaggedErrorClass()( "WorkOSError", diff --git a/apps/cloud/src/auth/handlers.ts b/apps/cloud/src/auth/handlers.ts index fca2cf6a5..05bc5ec17 100644 --- a/apps/cloud/src/auth/handlers.ts +++ b/apps/cloud/src/auth/handlers.ts @@ -22,6 +22,7 @@ import { env } from "cloudflare:workers"; import { WorkOSError } from "./errors"; import { WorkOSClient } from "./workos"; import { AutumnService } from "../extensions/billing/service"; +import { captureCauseEffect } from "../observability"; import { hasPaidOrganizationSubscription, isOverFreeOrganizationLimit, @@ -144,8 +145,8 @@ const deleteResponseCookie = (response: HttpServerResponse.HttpServerResponse, n HttpServerResponse.setCookieUnsafe(response, name, "", DELETE_COOKIE_OPTIONS); // --------------------------------------------------------------------------- -// Single non-protected API surface — public (login/callback) + session -// (me/logout/organizations/switch-organization). The session group has SessionAuth on it. +// Single non-protected API surface — public (login/callback/logout) + session +// (me/organizations/switch-organization). The session group has SessionAuth on it. // --------------------------------------------------------------------------- export const NonProtectedApi = HttpApi.make("cloudWeb").add(CloudAuthPublicApi).add(CloudAuthApi); @@ -202,7 +203,7 @@ export const CloudAuthPublicHandlers = HttpApiBuilder.group( const result = yield* workos.authenticateWithCode(query.code); // Mirror the account locally - yield* users.use((s) => s.ensureAccount(result.user.id)); + yield* users.use("ensureAccount", (s) => s.ensureAccount(result.user.id)); let sealedSession = result.sealedSession; @@ -274,6 +275,42 @@ export const CloudAuthPublicHandlers = HttpApiBuilder.group( ); }), ) + .handleRaw("logout", ({ request }) => + Effect.gen(function* () { + const workos = yield* WorkOSClient; + // The session this browser presents, NOT one the middleware vouched + // for — signing out of a session that has already ended must still + // sign the browser out (see the group declaration in ./api.ts). + const sealedSession = request.cookies["wos-session"] ?? ""; + + // WorkOS's documented sign-out: send the browser through the WorkOS + // logout endpoint, which ends the AuthKit session upstream and then + // redirects to the registered sign-out URL. Without this hop, the + // hosted session survives and the next "Sign in" silently + // re-authenticates (issue #1445). Fail-open when the cookie won't + // unseal — there is then nothing to end upstream, and local sign-out + // must still complete, so fall back to "/". + const origin = env.VITE_PUBLIC_SITE_URL ?? ""; + const logoutUrl = sealedSession + ? yield* workos.logoutUrl(sealedSession, origin ? `${origin}/` : undefined) + : null; + + const response = HttpServerResponse.redirect(logoutUrl ?? "/", { status: 302 }); + + // Drop only what this browser actually presented. Both cookies are + // SameSite=Lax, so a cross-site form POST carries neither — it gets + // the bare redirect and cannot be used to sign anyone out. + if (!sealedSession && request.cookies[AUTH_HINT_COOKIE] === undefined) return response; + + // The auth-hint travels with the session: leaving it behind would + // make the next page load optimistically paint the app shell for a + // signed-out browser. + return deleteResponseCookie( + deleteResponseCookie(response, "wos-session"), + AUTH_HINT_COOKIE, + ); + }), + ) // CLI device-login discovery. The WorkOS device endpoints live on the // WorkOS API host (`WORKOS_API_URL`, or api.workos.com in production, // the SAME base the SDK uses, so e2e points the CLI at the emulator with @@ -319,35 +356,6 @@ export const CloudSessionAuthHandlers = HttpApiBuilder.group( }; }), ) - .handleRaw("logout", () => - Effect.gen(function* () { - const workos = yield* WorkOSClient; - const session = yield* SessionContext; - - // WorkOS's documented sign-out: send the browser through the WorkOS - // logout endpoint, which ends the AuthKit session upstream and then - // redirects to the registered sign-out URL. Without this hop, the - // hosted session survives and the next "Sign in" silently - // re-authenticates (issue #1445). Fail-open when the cookie won't - // unseal: local sign-out must still complete, so fall back to "/". - const origin = env.VITE_PUBLIC_SITE_URL ?? ""; - const logoutUrl = yield* workos.logoutUrl( - session.sealedSession, - origin ? `${origin}/` : undefined, - ); - - // The auth-hint travels with the session: leaving it behind would - // make the next page load optimistically paint the app shell for a - // signed-out browser. - return deleteResponseCookie( - deleteResponseCookie( - HttpServerResponse.redirect(logoutUrl ?? "/", { status: 302 }), - "wos-session", - ), - AUTH_HINT_COOKIE, - ); - }), - ) .handle("organizations", () => Effect.gen(function* () { const workos = yield* WorkOSClient; @@ -408,7 +416,9 @@ export const CloudSessionAuthHandlers = HttpApiBuilder.group( ), { concurrency: 3 }, ).pipe( - Effect.catchTag("AutumnError", () => Effect.fail(new WorkOSError())), + // Any Autumn failure here (outage or missing customer) leaves the + // paid/free split unknown, and the limit must fail closed. + Effect.mapError(() => new WorkOSError()), Effect.map((ids) => new Set(ids.filter(Predicate.isNotNull))), ); @@ -420,10 +430,28 @@ export const CloudSessionAuthHandlers = HttpApiBuilder.group( const org = yield* workos.createOrganization(name); yield* workos.createMembership(org.id, session.accountId, "admin"); // `upsertOrganization` mints the slug at insert — no separate heal step. - const mirrored = yield* users.use((s) => + const mirrored = yield* users.use("upsertOrganization", (s) => s.upsertOrganization({ id: org.id, name: org.name }), ); + // Provision the org's billing customer while we're the ones creating + // the org. Without this the first billing call an org ever makes is a + // non-creating one (balance check / usage track), which 404s and keeps + // 404ing — unlimited unbilled executions. Non-fatal: a billing blip + // must not block signup, and the billing seam heals a customer that + // is still missing later. + yield* autumn.ensureCustomer(org.id).pipe( + Effect.catch((error) => + Effect.gen(function* () { + yield* Effect.logWarning( + "createOrganization: could not provision the Autumn customer", + { organizationId: org.id, error }, + ); + yield* captureCauseEffect(error); + }), + ), + ); + // Try to attach the new org to the current session. This can fail // (or silently return a session still scoped to the old org) when // the caller's current session is stale — most commonly after the @@ -474,7 +502,7 @@ export const CloudSessionAuthHandlers = HttpApiBuilder.group( // The typed confirmation must match the org's current name — the same // label the settings page shows. Trimmed on both sides. - const org = yield* users.use((s) => s.getOrganization(organizationId)); + const org = yield* users.use("getOrganization", (s) => s.getOrganization(organizationId)); if (!org || payload.confirmName.trim() !== org.name.trim()) { return yield* new OrganizationDeletionForbidden(); } @@ -492,7 +520,7 @@ export const CloudSessionAuthHandlers = HttpApiBuilder.group( // everyone (unreachable) but its secrets/tenant rows linger orphaned — // alert loudly so that window gets swept, then surface the failure. yield* users - .use((s) => s.deleteOrganizationCascade(organizationId)) + .use("deleteOrganizationCascade", (s) => s.deleteOrganizationCascade(organizationId)) .pipe( Effect.tapError((error) => Effect.logError( @@ -508,7 +536,9 @@ export const CloudSessionAuthHandlers = HttpApiBuilder.group( yield* autumn .use((client) => client.customers.delete({ customerId: organizationId })) .pipe( - Effect.catchTag("AutumnError", (error) => + // Includes the "customer never existed" answer: nothing to cancel + // is a fine outcome for a deleted org, and it is still worth a line. + Effect.catch((error) => Effect.logWarning("deleteOrganization: failed to delete Autumn customer", { organizationId, error, @@ -593,7 +623,7 @@ export const CloudSessionAuthHandlers = HttpApiBuilder.group( // Mirror the org locally so domain tables can FK against it; the // upsert mints the slug at insert — no separate heal step. const org = yield* workos.getOrganization(invitation.organizationId); - const mirrored = yield* users.use((s) => + const mirrored = yield* users.use("upsertOrganization", (s) => s.upsertOrganization({ id: org.id, name: org.name }), ); diff --git a/apps/cloud/src/auth/jwks-cache.node.test.ts b/apps/cloud/src/auth/jwks-cache.node.test.ts index d1f2b47c4..f4f3e08a6 100644 --- a/apps/cloud/src/auth/jwks-cache.node.test.ts +++ b/apps/cloud/src/auth/jwks-cache.node.test.ts @@ -9,7 +9,7 @@ import { type KeyLike, } from "jose"; -import { createCachedRemoteJWKSet } from "./jwks-cache"; +import { createCachedRemoteJWKSet, type JwksStore, type StoredJwks } from "./jwks-cache"; const issuer = "https://test-authkit.example.com"; const audience = "client_test_fixture"; @@ -65,6 +65,40 @@ const makeFetchHarness = (initialKeys: ReadonlyArray): FetchHarness => { }; }; +interface StoreHarness extends JwksStore { + readonly seed: (stored: StoredJwks) => void; + readonly reads: () => number; + readonly writes: () => number; +} + +/** Stands in for the Workers Cache API: shared across "isolates", in memory. */ +const makeStoreHarness = (): StoreHarness => { + const entries = new Map(); + let reads = 0; + let writes = 0; + return { + get: async (url) => { + reads++; + return entries.get(url.toString()) ?? null; + }, + put: async (url, stored) => { + writes++; + entries.set(url.toString(), stored); + }, + seed: (stored) => { + entries.set(jwksUrl.toString(), stored); + }, + reads: () => reads, + writes: () => writes, + }; +}; + +/** A fetch that always fails, standing in for a slow/down key server. */ +const failingFetch: typeof globalThis.fetch = async () => { + // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- test double: `fetch` signals an unreachable key server by rejecting + throw new Error("JWKS endpoint unreachable"); +}; + describe("createCachedRemoteJWKSet", () => { it("FAILING-WITHOUT-CACHE: N verifications hit JWKS endpoint only once within TTL", async () => { const kp = await generateRotatableKeypair("k1"); @@ -168,4 +202,151 @@ describe("createCachedRemoteJWKSet", () => { await jwtVerify(t1, jwks, { issuer, audience }); expect(harness.callCount()).toBe(2); }); + + // ------------------------------------------------------------------------- + // Cross-isolate store — the cold-isolate path that caused the 2026-08-18 + // 500s: no module-scope entry, so every verify paid an upstream fetch. + // ------------------------------------------------------------------------- + + it("a cold isolate serves from the store instead of going upstream", async () => { + const kp = await generateRotatableKeypair("k1"); + const store = makeStoreHarness(); + + // First isolate: cold everywhere, so it fetches and populates the store. + const warm = makeFetchHarness([kp.publicJwk]); + const first = createCachedRemoteJWKSet(jwksUrl, { fetch: warm.fetch, store }); + const t1 = await sign(kp); + await jwtVerify(t1, first, { issuer, audience }); + expect(warm.callCount()).toBe(1); + expect(store.writes()).toBe(1); + + // A brand-new isolate (fresh module scope). If it reaches upstream at all + // the fetch fails, so verifying proves the store answered. + const second = createCachedRemoteJWKSet(jwksUrl, { fetch: failingFetch, store }); + const { payload } = await jwtVerify(t1, second, { issuer, audience }); + expect(payload.sub).toBe("user_test"); + expect(store.reads()).toBeGreaterThan(0); + }); + + it("keeps serving the last good keys when the key server is down", async () => { + const kp = await generateRotatableKeypair("k1"); + const harness = makeFetchHarness([kp.publicJwk]); + let upstreamUp = true; + const flaky: typeof globalThis.fetch = (...args) => + upstreamUp ? harness.fetch(...args) : failingFetch(...args); + + // ttl short enough that the next call is past it, stale window generous. + const jwks = createCachedRemoteJWKSet(jwksUrl, { + fetch: flaky, + store: null, + ttlMs: 10, + staleMaxMs: 60_000, + }); + + const token = await sign(kp); + await jwtVerify(token, jwks, { issuer, audience }); + + upstreamUp = false; + await new Promise((r) => setTimeout(r, 20)); + + // Past the TTL with a dead upstream: the cached keys still verify. + const { payload } = await jwtVerify(token, jwks, { issuer, audience }); + expect(payload.sub).toBe("user_test"); + }); + + it("stops serving stale keys once the stale window closes", async () => { + const kp = await generateRotatableKeypair("k1"); + const harness = makeFetchHarness([kp.publicJwk]); + let upstreamUp = true; + const flaky: typeof globalThis.fetch = (...args) => + upstreamUp ? harness.fetch(...args) : failingFetch(...args); + + const jwks = createCachedRemoteJWKSet(jwksUrl, { + fetch: flaky, + store: null, + ttlMs: 5, + staleMaxMs: 10, + }); + + const token = await sign(kp); + await jwtVerify(token, jwks, { issuer, audience }); + + upstreamUp = false; + await new Promise((r) => setTimeout(r, 30)); + + // Beyond staleMaxMs the keys are no longer trustworthy — fail, don't + // silently honour a key set we can no longer confirm. + await expect(jwtVerify(token, jwks, { issuer, audience })).rejects.toThrow(); + }); + + it("attributes background revalidation to fetchCount but not blockingFetchCount", async () => { + const kp = await generateRotatableKeypair("k1"); + const harness = makeFetchHarness([kp.publicJwk]); + const jwks = createCachedRemoteJWKSet(jwksUrl, { + fetch: harness.fetch, + store: null, + ttlMs: 10, + staleMaxMs: 60_000, + }); + + const token = await sign(kp); + await jwtVerify(token, jwks, { issuer, audience }); + expect(jwks.inspect().blockingFetchCount).toBe(1); + + await new Promise((r) => setTimeout(r, 20)); + + // Past the TTL: served from the stale entry, revalidated behind it. The + // caller waited on nothing, so latency must not be attributed to it. + const before = jwks.inspect(); + await jwtVerify(token, jwks, { issuer, audience }); + const after = jwks.inspect(); + + expect(after.blockingFetchCount).toBe(before.blockingFetchCount); + expect(after.fetchCount).toBeGreaterThan(before.fetchCount); + }); + + it("records a cold-isolate store read as a store hit, not a fetch", async () => { + const kp = await generateRotatableKeypair("k1"); + const store = makeStoreHarness(); + const warm = makeFetchHarness([kp.publicJwk]); + + const first = createCachedRemoteJWKSet(jwksUrl, { fetch: warm.fetch, store }); + const token = await sign(kp); + await jwtVerify(token, first, { issuer, audience }); + + const second = createCachedRemoteJWKSet(jwksUrl, { fetch: failingFetch, store }); + await jwtVerify(token, second, { issuer, audience }); + + const stats = second.inspect(); + expect(stats.storeHitCount).toBe(1); + expect(stats.blockingFetchCount).toBe(0); + }); + + it("does not block a request on revalidation once keys are cached", async () => { + const kp = await generateRotatableKeypair("k1"); + const harness = makeFetchHarness([kp.publicJwk]); + let hang = false; + const slowAfterFirst: typeof globalThis.fetch = async (...args) => { + if (hang) await new Promise((r) => setTimeout(r, 5_000)); + return harness.fetch(...args); + }; + + const jwks = createCachedRemoteJWKSet(jwksUrl, { + fetch: slowAfterFirst, + store: null, + ttlMs: 10, + staleMaxMs: 60_000, + }); + + const token = await sign(kp); + await jwtVerify(token, jwks, { issuer, audience }); + + hang = true; + await new Promise((r) => setTimeout(r, 20)); + + // The revalidation behind this call hangs for 5s; the verify must not. + const startedAt = Date.now(); + await jwtVerify(token, jwks, { issuer, audience }); + expect(Date.now() - startedAt).toBeLessThan(1_000); + }); }); diff --git a/apps/cloud/src/auth/jwks-cache.ts b/apps/cloud/src/auth/jwks-cache.ts index d04e80187..3cd1d963d 100644 --- a/apps/cloud/src/auth/jwks-cache.ts +++ b/apps/cloud/src/auth/jwks-cache.ts @@ -1,5 +1,5 @@ // --------------------------------------------------------------------------- -// In-memory JWKS cache for MCP JWT verification. +// JWKS cache for JWT verification (session + MCP auth). // --------------------------------------------------------------------------- // // Cloudflare Workers boot many short-lived isolates. `createRemoteJWKSet`'s @@ -7,14 +7,32 @@ // fetches per hour because each new isolate starts cold. Production p99 for // `mcp.auth.jwt_verify` was 1.7s — almost entirely the JWKS fetch. // -// This module offers a drop-in `createCachedRemoteJWKSet` that: +// Module-scope memory alone only helps while an isolate stays warm. When +// `workos.session.local_verify` started missing ~92% of the time (2026-08-18), +// every miss paid a fresh upstream fetch — p50 3s, and the tail crossed the 5s +// request timeout, turning a slow key-server into `AbortError` → 500 on every +// authenticated API route and 503 on `/mcp`. A transient key-server blip must +// never read as an auth failure, and a cold isolate must not have to go +// upstream at all. // -// * Caches the JSON Web Key Set in module-scope memory for a configurable -// TTL (default 1 hour). -// * Single-flights concurrent fetches so a stampede of verifies during a -// cache miss only fires one upstream request. -// * Force-refreshes once when verification fails with a cached key, so -// genuine key rotation isn't blocked by the TTL. +// So this module layers three caches, fastest first: +// +// 1. Module-scope memory — free, but dies with the isolate. +// 2. A cross-isolate store (the Workers Cache API by default) — colo-local, +// survives isolate recycling, so a cold isolate reads keys in ~1ms +// instead of paying an upstream round trip. +// 3. The upstream JWKS endpoint. +// +// On top of that it is stale-while-revalidate: once past `ttlMs` a usable key +// set is served immediately and refreshed in the background, and if a refresh +// fails we keep serving the last good keys until `staleMaxMs`. Only a fully +// cold path (no memory, no store) ever blocks on the network. +// +// Serving stale keys is safe in a way that serving a stale *token* would not +// be: key sets rotate on the order of days, tokens are still signature- and +// expiry-checked against them, and a token whose `kid` is absent forces a real +// refresh before it is rejected (see `get`). `staleMaxMs` bounds how long a +// retired key can still be honoured. // // The returned function is a `JWTVerifyGetKey` and slots directly into // `jose.jwtVerify`. It also exposes `forceRefresh()` so the verify path can @@ -32,6 +50,21 @@ import { import { Schema } from "effect"; import { JWKSNoMatchingKey } from "jose/errors"; +/** + * A cross-isolate key-set store. Defaults to the Workers Cache API; tests and + * non-Workers hosts pass their own, or `null` to stay memory-only. + */ +export interface JwksStore { + readonly get: (url: URL) => Promise; + readonly put: (url: URL, stored: StoredJwks) => Promise; +} + +export interface StoredJwks { + readonly jwks: JSONWebKeySet; + /** Epoch ms of the upstream fetch these keys came from. */ + readonly fetchedAt: number; +} + export interface CachedRemoteJWKSetOptions { /** * How long a successful fetch is considered fresh. Defaults to 1 hour — @@ -39,20 +72,56 @@ export interface CachedRemoteJWKSetOptions { * failure handles unscheduled rotations. */ readonly ttlMs?: number; + /** + * How long past `ttlMs` a key set may still be served while refreshes are + * failing. Defaults to 24h — long enough to ride out a key-server outage, + * short enough to bound how long a retired key stays honoured. + */ + readonly staleMaxMs?: number; /** Override the fetch implementation for tests. */ readonly fetch?: typeof globalThis.fetch; /** HTTP request timeout. Defaults to 5s, matching jose. */ readonly timeoutMs?: number; + /** + * Cross-isolate store. Defaults to the Workers Cache API when available, + * `null` disables the layer (memory-only). + */ + readonly store?: JwksStore | null; } export interface CachedRemoteJWKSet extends JWTVerifyGetKey { /** Drop the cached JWKS so the next call refetches. */ readonly forceRefresh: () => void; - /** Inspect the current cache state (testing/diagnostics). */ - readonly inspect: () => { fetchedAt: number | null; hasJwks: boolean }; + /** + * Inspect the current cache state (testing/diagnostics/span annotation). + * `fetchCount`/`fetchFailureCount` are lifetime counters for this resolver + * instance; callers snapshot them around a verify to tell a cache hit from + * a live upstream fetch (the Aug 2026 latency regression was exactly this + * cache silently missing on ~92% of verifies, invisible in traces). + * + * `blockingFetchCount` counts only fetches a verify actually WAITED on. + * Under stale-while-revalidate `fetchCount` also moves for background + * revalidation the caller never paid for, so latency attribution wants + * `blockingFetchCount`. `storeHitCount` counts cold-isolate reads answered + * by the cross-isolate store instead of going upstream. + */ + readonly inspect: () => { + fetchedAt: number | null; + hasJwks: boolean; + fetchCount: number; + fetchFailureCount: number; + blockingFetchCount: number; + storeHitCount: number; + lastFetchDurationMs: number | null; + /** Wall-clock of the last cross-isolate store read (I/O). */ + lastStoreReadMs: number | null; + /** Wall-clock of the last key resolution — WebCrypto `importKey`. */ + lastResolveMs: number | null; + }; } const DEFAULT_TTL_MS = 60 * 60 * 1000; +const DEFAULT_STALE_MAX_MS = 24 * 60 * 60 * 1000; const DEFAULT_TIMEOUT_MS = 5000; const JsonWebKey = Schema.Record(Schema.String, Schema.Unknown); @@ -73,6 +142,26 @@ interface CacheEntry { resolver: (protectedHeader: JWTHeaderParameters, token?: FlattenedJWSInput) => Promise; } +const entryFrom = (stored: StoredJwks): CacheEntry => ({ + jwks: stored.jwks, + fetchedAt: stored.fetchedAt, + resolver: createLocalJWKSet(stored.jwks), +}); + +/** + * Cache upkeep (store writes, background revalidation) is best effort: it must + * never fail or delay the verify that happened to trigger it. + */ +const ignoreFailure = async (work: Promise): Promise => { + // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: best-effort cache upkeep must not surface as a verify failure + try { + await work; + } catch { + // Deliberately swallowed — the caller already has usable keys, or will + // take the upstream path on its next miss. + } +}; + const fetchJwksOnce = async ( url: URL, fetchImpl: typeof globalThis.fetch, @@ -107,17 +196,74 @@ const fetchJwksOnce = async ( } }; +// --------------------------------------------------------------------------- +// Workers Cache API store +// --------------------------------------------------------------------------- +// +// Keyed by the JWKS URL itself. We store our own Response rather than the +// upstream one so the body is already-validated JSON and the cache headers are +// ours: `max-age` covers the stale window, because an entry past `ttlMs` is +// still useful to us as a stale fallback. + +const STORE_HEADER_FETCHED_AT = "x-jwks-fetched-at"; + +/** `caches.default` is a Workers extension the standard lib type omits. */ +type WorkersCacheStorage = CacheStorage & { readonly default?: Cache }; + +const workersCacheStore = (staleMaxMs: number): JwksStore | null => { + if (typeof caches === "undefined") return null; + const open = (): Cache | null => (caches as WorkersCacheStorage).default ?? null; + + return { + get: async (url) => { + const cache = open(); + if (!cache) return null; + const hit = await cache.match(url.toString()); + if (!hit) return null; + const fetchedAtHeader = hit.headers.get(STORE_HEADER_FETCHED_AT); + const fetchedAt = fetchedAtHeader === null ? Number.NaN : Number(fetchedAtHeader); + if (!Number.isFinite(fetchedAt)) return null; + const body = await hit.json(); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: a corrupt cache entry must degrade to a miss, never fail the verify + try { + await decodeJsonWebKeySetPayload(body); + } catch { + return null; + } + return { jwks: body as JSONWebKeySet, fetchedAt }; + }, + put: async (url, stored) => { + const cache = open(); + if (!cache) return; + const maxAgeSeconds = Math.max(1, Math.floor(staleMaxMs / 1000)); + await cache.put( + url.toString(), + new Response(JSON.stringify(stored.jwks), { + status: 200, + headers: { + "content-type": "application/json", + "cache-control": `max-age=${maxAgeSeconds}`, + [STORE_HEADER_FETCHED_AT]: String(stored.fetchedAt), + }, + }), + ); + }, + }; +}; + /** * Creates a cached, single-flight, force-refreshable JWKS resolver compatible * with `jose.jwtVerify`. Drop-in replacement for `createRemoteJWKSet` for the - * MCP auth path — see module header for why we don't just use jose's built-in. + * auth paths — see module header for why we don't just use jose's built-in. */ export const createCachedRemoteJWKSet = ( url: URL, options: CachedRemoteJWKSetOptions = {}, ): CachedRemoteJWKSet => { const ttlMs = options.ttlMs ?? DEFAULT_TTL_MS; + const staleMaxMs = Math.max(options.staleMaxMs ?? DEFAULT_STALE_MAX_MS, ttlMs); const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS; + const store = options.store === undefined ? workersCacheStore(staleMaxMs) : options.store; // Capture the fetch impl lazily so consumers can swap globalThis.fetch // (tests do this) without us snapshotting a stale reference. const fetchImpl = (): typeof globalThis.fetch => @@ -125,39 +271,116 @@ export const createCachedRemoteJWKSet = ( let entry: CacheEntry | null = null; let inflight: Promise | null = null; + let fetchCount = 0; + let fetchFailureCount = 0; + let blockingFetchCount = 0; + let storeHitCount = 0; + let lastFetchDurationMs: number | null = null; + let lastStoreReadMs: number | null = null; + let lastResolveMs: number | null = null; + + const isFresh = (candidate: CacheEntry): boolean => Date.now() - candidate.fetchedAt < ttlMs; + const isUsable = (candidate: CacheEntry): boolean => + Date.now() - candidate.fetchedAt < staleMaxMs; const refresh = (): Promise => { if (inflight) return inflight; + const startedAt = Date.now(); + fetchCount += 1; inflight = (async () => { const jwks = await fetchJwksOnce(url, fetchImpl(), timeoutMs); - const next: CacheEntry = { - jwks, - fetchedAt: Date.now(), - resolver: createLocalJWKSet(jwks), - }; + const next = entryFrom({ jwks, fetchedAt: Date.now() }); entry = next; + if (store) { + await ignoreFailure(store.put(url, { jwks: next.jwks, fetchedAt: next.fetchedAt })); + } return next; - })().finally(() => { - inflight = null; - }); + })() + .then( + (next) => next, + (error: unknown) => { + fetchFailureCount += 1; + // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: counting a fetch failure must preserve the original rejection for jose + throw error; + }, + ) + .finally(() => { + lastFetchDurationMs = Date.now() - startedAt; + inflight = null; + }); return inflight; }; - const ensureFresh = async (forceRefresh: boolean): Promise => { - if (forceRefresh) return refresh(); - if (entry && Date.now() - entry.fetchedAt < ttlMs) return entry; + /** Fire-and-forget revalidation behind a stale hit. */ + const refreshInBackground = (): void => { + void ignoreFailure(refresh()); + }; + + const loadFromStore = async (): Promise => { + if (!store) return null; + // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: the L2 store is an optimization; any failure degrades to an upstream fetch + try { + const storeReadStartedAt = Date.now(); + const stored = await store.get(url); + lastStoreReadMs = Date.now() - storeReadStartedAt; + if (!stored) return null; + const candidate = entryFrom(stored); + if (!isUsable(candidate)) return null; + entry = candidate; + storeHitCount += 1; + return candidate; + } catch { + return null; + } + }; + + /** A refresh the caller waits on — the only kind that costs it latency. */ + const refreshBlocking = (): Promise => { + blockingFetchCount += 1; return refresh(); }; + const ensureFresh = async (forceRefresh: boolean): Promise => { + if (forceRefresh) return refreshBlocking(); + if (entry && isFresh(entry)) return entry; + + // Memory is stale but still usable: serve it now, revalidate behind it. + if (entry && isUsable(entry)) { + refreshInBackground(); + return entry; + } + + // Cold isolate — the L2 store saves us the upstream round trip. + const stored = await loadFromStore(); + if (stored) { + if (!isFresh(stored)) refreshInBackground(); + return stored; + } + + // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: a failed refresh must fall back to stale keys rather than fail the verify + try { + return await refreshBlocking(); + } catch (error) { + // Upstream is slow or down. Last good keys beat failing every request. + if (entry && isUsable(entry)) return entry; + // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: nothing usable is cached, so the upstream failure is the real answer + throw error; + } + }; + const get: JWTVerifyGetKey = async (protectedHeader, token) => { const current = await ensureFresh(false); // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: jose JWTVerifyGetKey retry path is defined by thrown resolver failures try { - return await current.resolver(protectedHeader, token); + const resolveStartedAt = Date.now(); + const key = await current.resolver(protectedHeader, token); + lastResolveMs = Date.now() - resolveStartedAt; + return key; } catch (error) { - // Likely cause: keys rotated upstream after our TTL window started. - // Refetch once and try again. Anything still failing bubbles up so - // jose can classify it (we do not silently swallow real failures). + // Likely cause: keys rotated upstream after our TTL window started, or + // we answered from a stale entry. Refetch once and try again. Anything + // still failing bubbles up so jose can classify it (we do not silently + // swallow real failures). if (!isJwksNoMatchingKey(error)) { // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: jose JWTVerifyGetKey requires preserving upstream resolver rejection throw error; @@ -177,6 +400,13 @@ export const createCachedRemoteJWKSet = ( value: () => ({ fetchedAt: entry?.fetchedAt ?? null, hasJwks: entry !== null, + fetchCount, + fetchFailureCount, + blockingFetchCount, + storeHitCount, + lastFetchDurationMs, + lastStoreReadMs, + lastResolveMs, }), }); return result; diff --git a/apps/cloud/src/auth/last-org-cookie.ts b/apps/cloud/src/auth/last-org-cookie.ts index dde7c9284..ef6242604 100644 --- a/apps/cloud/src/auth/last-org-cookie.ts +++ b/apps/cloud/src/auth/last-org-cookie.ts @@ -8,7 +8,7 @@ // cookie fills the gap: the client records the slug of the org it's verifiably // viewing, and the two bare-entry deciders honor it — // -// - the SSR auth gate redirects bare document paths onto it (ssr-gate.ts) +// - the document auth gate redirects bare document paths onto it (doc-gate.ts) // - the login callback prefers it when picking the org for a fresh session // with a bare returnTo (handlers.ts) // diff --git a/apps/cloud/src/auth/org-api-key-auth.node.test.ts b/apps/cloud/src/auth/org-api-key-auth.node.test.ts index 1d1e00b9a..ec87511c3 100644 --- a/apps/cloud/src/auth/org-api-key-auth.node.test.ts +++ b/apps/cloud/src/auth/org-api-key-auth.node.test.ts @@ -62,7 +62,7 @@ const stubWorkOS = Layer.succeed( ); const stubUsers = Layer.succeed(UserStoreService)({ - use: (fn) => + use: (_op, fn) => Effect.promise(() => fn({ ensureAccount: async (id: string) => ({ id, createdAt }), diff --git a/apps/cloud/src/auth/org-selector-auth.node.test.ts b/apps/cloud/src/auth/org-selector-auth.node.test.ts index 5c59ba7de..ead56fb89 100644 --- a/apps/cloud/src/auth/org-selector-auth.node.test.ts +++ b/apps/cloud/src/auth/org-selector-auth.node.test.ts @@ -64,7 +64,7 @@ const stubWorkOS = Layer.succeed( ); const stubUsers = Layer.succeed(UserStoreService)({ - use: (fn) => + use: (_op, fn) => Effect.promise(() => fn({ ensureAccount: async (id: string) => ({ id, createdAt }), diff --git a/apps/cloud/src/auth/organization.ts b/apps/cloud/src/auth/organization.ts index 51944cfbb..073dfacb3 100644 --- a/apps/cloud/src/auth/organization.ts +++ b/apps/cloud/src/auth/organization.ts @@ -38,12 +38,14 @@ import { WorkOSClient } from "./workos"; export const resolveOrganization = (organizationId: string) => Effect.gen(function* () { const users = yield* UserStoreService; - const existing = yield* users.use((s) => s.getOrganization(organizationId)); + const existing = yield* users.use("getOrganization", (s) => s.getOrganization(organizationId)); if (existing) return existing; const workos = yield* WorkOSClient; const fresh = yield* workos.getOrganization(organizationId); - return yield* users.use((s) => s.upsertOrganization({ id: fresh.id, name: fresh.name })); + return yield* users.use("upsertOrganization", (s) => + s.upsertOrganization({ id: fresh.id, name: fresh.name }), + ); }); // --------------------------------------------------------------------------- @@ -125,7 +127,7 @@ export const authorizeOrganizationSelector = (userId: string, selector: string) return yield* authorizeOrganization(userId, selector); } const users = yield* UserStoreService; - const org = yield* users.use((s) => s.getOrganizationBySlug(selector)); + const org = yield* users.use("getOrganizationBySlug", (s) => s.getOrganizationBySlug(selector)); if (!org) return null; return yield* authorizeOrganization(userId, org.id); }); diff --git a/apps/cloud/src/auth/user-store-error.node.test.ts b/apps/cloud/src/auth/user-store-error.node.test.ts new file mode 100644 index 000000000..880c6fa37 --- /dev/null +++ b/apps/cloud/src/auth/user-store-error.node.test.ts @@ -0,0 +1,128 @@ +// `UserStoreError` is the public failure of every cloud user-store call, and it +// used to carry NOTHING: no operation, no reason, no message. Sentry grouped +// every store failure — a connect timeout, a missing table, a constraint +// violation — into one titleless issue, and the only cause detail (the +// pretty-printed Effect cause stuffed into a Sentry `extra`) is scrubbed +// server-side, so the issue was undiagnosable from Sentry alone. +// +// This pins the two safe classification fields it must carry instead: +// `operation` (already in hand at the call site) and `reason` (classified from +// the driver cause the way `statusFromWorkOSCause` classifies WorkOS causes). +import { createServer, type Server, type Socket } from "node:net"; + +import { describe, expect, it } from "@effect/vitest"; +import { Effect, Layer, Result } from "effect"; +import postgres from "postgres"; + +import { UserStoreService } from "./context"; +import { ServiceAdapterError, userStoreReasonFromCause, type UserStoreError } from "./errors"; +import { DbService } from "../db/db"; + +// A socket that completes the TCP handshake and then says nothing — exactly +// what a wedged Hyperdrive/Postgres endpoint looks like to postgres.js, and the +// only way to obtain the driver's REAL connect-timeout error object rather than +// a hand-written imitation of it. +const blackHolePort = async (): Promise<{ + readonly port: number; + readonly close: () => Promise; +}> => { + const sockets = new Set(); + const server: Server = createServer((socket) => { + sockets.add(socket); + socket.on("close", () => sockets.delete(socket)); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + // oxlint-disable-next-line executor/no-error-constructor, executor/no-try-catch-or-throw -- boundary: the fixture cannot run without a bound port + if (address === null || typeof address === "string") throw new Error("no port"); + return { + port: address.port, + close: () => + new Promise((resolve) => { + // The timed-out client leaves its half-open socket attached; without + // dropping it first, `close` waits for a peer that will never speak. + for (const socket of sockets) socket.destroy(); + server.close(() => resolve()); + }), + }; +}; + +const realConnectTimeoutError = async (): Promise => { + const hole = await blackHolePort(); + const sql = postgres(`postgresql://postgres:postgres@127.0.0.1:${hole.port}/postgres`, { + max: 1, + connect_timeout: 1, + fetch_types: false, + onnotice: () => undefined, + }); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: capturing the driver's own thrown error IS the fixture + try { + await sql`select 1`; + // oxlint-disable-next-line executor/no-error-constructor, executor/no-try-catch-or-throw -- boundary: the fixture is unusable if the socket answered + throw new Error("expected the connect to time out"); + } catch (error) { + return error; + } finally { + // oxlint-disable-next-line executor/no-promise-catch -- boundary: best-effort teardown of a connection that never opened + await sql.end({ timeout: 0 }).catch(() => undefined); + await hole.close(); + } +}; + +// Drizzle re-throws driver failures wrapped in its own error with the failing +// SQL in the message and the driver error in `.cause` — the shape production +// actually reports (`Failed query: select … -> write CONNECT_TIMEOUT …`). +const wrappedLikeDrizzle = (cause: unknown): Error => + // oxlint-disable-next-line executor/no-error-constructor -- boundary: reproducing the driver wrapper shape the store really fails with + Object.assign(new Error('Failed query: select "id" from "organizations" where "id" = $1'), { + cause, + }); + +const stubDb = Layer.succeed(DbService)({ db: {} as never }); + +const failingStoreCall = ( + operation: string, + failure: unknown, +): Effect.Effect> => + Effect.gen(function* () { + const users = yield* UserStoreService; + // oxlint-disable-next-line executor/no-promise-reject -- boundary: the store adapter lifts a REJECTING promise; rejecting is the fixture + return yield* users.use(operation, () => Promise.reject(failure)); + }).pipe( + Effect.provide(UserStoreService.Live.pipe(Layer.provide(stubDb))), + Effect.result, + ) as Effect.Effect>; + +describe("UserStoreError classification", () => { + it("classifies the driver cause chain", async () => { + const driverError = await realConnectTimeoutError(); + + expect(userStoreReasonFromCause(wrappedLikeDrizzle(driverError))).toBe("connect_timeout"); + expect( + userStoreReasonFromCause(new ServiceAdapterError({ cause: wrappedLikeDrizzle(driverError) })), + ).toBe("connect_timeout"); + expect( + userStoreReasonFromCause( + // oxlint-disable-next-line executor/no-error-constructor -- boundary: a SQLSTATE failure shape + Object.assign(new Error('relation "organizations" does not exist'), { code: "42P01" }), + ), + ).toBe("query"); + // oxlint-disable-next-line executor/no-error-constructor -- boundary: a failure with nothing to classify + expect(userStoreReasonFromCause(new Error("something else"))).toBe("unknown"); + }, 20_000); + + it("carries operation, reason and a stable message on the public error", async () => { + const driverError = await realConnectTimeoutError(); + + const result = await Effect.runPromise( + failingStoreCall("getOrganization", wrappedLikeDrizzle(driverError)), + ); + + expect(Result.isFailure(result)).toBe(true); + if (!Result.isFailure(result)) return; + expect(result.failure.operation).toBe("getOrganization"); + expect(result.failure.reason).toBe("connect_timeout"); + expect(result.failure.message).toContain("getOrganization"); + expect(result.failure.message).toContain("connect_timeout"); + }, 20_000); +}); diff --git a/apps/cloud/src/auth/workos.ts b/apps/cloud/src/auth/workos.ts index 88281115a..bf9130e8a 100644 --- a/apps/cloud/src/auth/workos.ts +++ b/apps/cloud/src/auth/workos.ts @@ -213,21 +213,82 @@ const verifySealedSessionLocally = ( jwks: CachedRemoteJWKSet, ): Effect.Effect => Effect.gen(function* () { + // Phase timings, not just child spans. `local_verify` is a leaf in + // production traces, so a ~3.3s verify has nothing under it to blame — + // and it stayed 3.3s after the JWKS fetch was eliminated entirely + // (jwks.fetch_count == 0), so the cost is one of the phases below. Under + // workerd `Date.now()` only advances at I/O boundaries, which is exactly + // what makes a raw span duration misleading here: recording each phase + // explicitly says which await the wall-clock actually crossed. + const verifyStartedAt = Date.now(); + + const unsealStartedAt = Date.now(); const unsealed = yield* Effect.tryPromise({ try: () => unsealWorkOSSession(sessionData, cookiePassword), catch: (cause) => new LocalSessionCookieError({ cause }), }).pipe( Effect.catchTag("LocalSessionCookieError", () => Effect.succeed(null as unknown | null)), + Effect.withSpan("workos.session.unseal"), ); + const unsealMs = Date.now() - unsealStartedAt; + yield* Effect.annotateCurrentSpan({ "verify.unseal_ms": unsealMs }); if (!unsealed) return { _tag: "InvalidCookie" }; + const decodeStartedAt = Date.now(); const session = Option.match(decodeSealedSessionPayload(unsealed), { onNone: (): SealedSessionPayload | null => null, onSome: (payload) => payload, }); + yield* Effect.annotateCurrentSpan({ "verify.decode_ms": Date.now() - decodeStartedAt }); if (!session) return { _tag: "InvalidCookie" }; - const verified = yield* verifyJwtWithRefreshRetry(session.accessToken, jwks); + // Snapshot the JWKS cache around the verify so the `local_verify` span + // says whether THIS verify was a warm-cache signature check or paid for a + // live upstream JWKS fetch. The Aug 2026 latency regression was the cache + // silently missing on most verifies, and no span attribute distinguished + // the two paths. + const jwksBefore = jwks.inspect(); + // Entry-state annotation goes on BEFORE the verify so a failing verify + // (the case worth debugging) still records whether the cache was warm. + yield* Effect.annotateCurrentSpan({ + "jwks.cache_populated_at_start": jwksBefore.hasJwks, + ...(jwksBefore.fetchedAt === null + ? {} + : { "jwks.cache_age_ms": Date.now() - jwksBefore.fetchedAt }), + }); + const jwtStartedAt = Date.now(); + const verified = yield* verifyJwtWithRefreshRetry(session.accessToken, jwks).pipe( + Effect.withSpan("workos.session.jwt_verify"), + Effect.onExit(() => { + const jwksAfter = jwks.inspect(); + const finishedAt = Date.now(); + return Effect.annotateCurrentSpan({ + "verify.jwt_ms": finishedAt - jwtStartedAt, + "verify.total_ms": finishedAt - verifyStartedAt, + // Blocking, not total: under stale-while-revalidate a background + // refresh moves `fetchCount` without costing this verify anything. + // Attribute latency to what the caller actually waited on. + "jwks.fetched_during_verify": + jwksAfter.blockingFetchCount > jwksBefore.blockingFetchCount, + "jwks.served_from_store": jwksAfter.storeHitCount > jwksBefore.storeHitCount, + "jwks.fetch_count": jwksAfter.fetchCount, + "jwks.blocking_fetch_count": jwksAfter.blockingFetchCount, + "jwks.fetch_failure_count": jwksAfter.fetchFailureCount, + ...(jwksAfter.lastFetchDurationMs === null + ? {} + : { "jwks.last_fetch_ms": jwksAfter.lastFetchDurationMs }), + // Splits the ~3.4s that sits inside jwt_verify with zero upstream + // fetches: the cross-isolate store read (I/O) vs WebCrypto key + // import vs the signature check itself. + ...(jwksAfter.lastStoreReadMs === null + ? {} + : { "jwks.store_read_ms": jwksAfter.lastStoreReadMs }), + ...(jwksAfter.lastResolveMs === null + ? {} + : { "jwks.key_resolve_ms": jwksAfter.lastResolveMs }), + }); + }), + ); if (!verified) return { _tag: "Refresh" }; const claims = Option.getOrNull(decodeJwtClaims(decodeJwt(session.accessToken))); @@ -338,9 +399,12 @@ const make = Effect.gen(function* () { // exception had one (all its typed exceptions do), so consumers can tell a // definitive WorkOS denial (401/403/404 — fail closed) from a transient // failure (429/5xx/network — retryable). - const use = (fn: (wos: WorkOS) => Promise) => + // `op` names the SDK call (mirroring its `namespace.method` path) so every + // span reads `workos.` instead of one undifferentiated "workos" + // bucket, and failures log which call actually failed. + const use = (op: string, fn: (wos: WorkOS) => Promise) => withServiceLogging( - "workos", + `workos.${op}`, workosErrorFromFailure, tryPromiseService(() => fn(workos)), ); @@ -376,7 +440,7 @@ const make = Effect.gen(function* () { if (isLocalSessionInvalidCookie(local)) return null; // Try refreshing - const refreshed = yield* use(() => session.refresh()).pipe( + const refreshed = yield* use("session.refresh", () => session.refresh()).pipe( Effect.orElseSucceed(() => ({ authenticated: false as const })), ); @@ -405,7 +469,7 @@ const make = Effect.gen(function* () { }), authenticateWithCode: (code: string) => - use((wos) => + use("userManagement.authenticateWithCode", (wos) => wos.userManagement.authenticateWithCode({ code, clientId, @@ -415,11 +479,13 @@ const make = Effect.gen(function* () { /** Create a new organization in WorkOS. */ createOrganization: (name: string) => - use((wos) => wos.organizations.createOrganization({ name })), + use("organizations.createOrganization", (wos) => + wos.organizations.createOrganization({ name }), + ), /** Add a user to an organization. */ createMembership: (organizationId: string, userId: string, roleSlug?: string) => - use((wos) => + use("userManagement.createOrganizationMembership", (wos) => wos.userManagement.createOrganizationMembership({ organizationId, userId, @@ -429,7 +495,7 @@ const make = Effect.gen(function* () { /** List organization memberships for a user. */ listUserMemberships: (userId: string) => - use(async (wos) => + use("userManagement.listOrganizationMemberships", async (wos) => collectWorkOSList( await wos.userManagement.listOrganizationMemberships({ userId, @@ -448,7 +514,7 @@ const make = Effect.gen(function* () { sessionData, cookiePassword, }); - const refreshed = yield* use(() => + const refreshed = yield* use("session.refresh", () => session.refresh(organizationId ? { organizationId } : undefined), ); if (!refreshed.authenticated || !("sealedSession" in refreshed)) return null; @@ -517,10 +583,13 @@ const make = Effect.gen(function* () { * auth/api-keys.ts. */ validateApiKey: (value: string) => - use((wos) => wos.apiKeys.validateApiKey({ value }) as Promise), + use( + "apiKeys.validateApiKey", + (wos) => wos.apiKeys.validateApiKey({ value }) as Promise, + ), listUserApiKeys: (userId: string, organizationId: string) => - use(async (wos) => { + use("userManagement.listUserApiKeys", async (wos) => { const raw = wos as RawWorkOS; return collectRawWorkOSList(async (after) => { const response = await raw.get(`/user_management/users/${userId}/api_keys`, { @@ -535,7 +604,7 @@ const make = Effect.gen(function* () { }), createUserApiKey: (params: { userId: string; organizationId: string; name: string }) => - use(async (wos) => { + use("userManagement.createUserApiKey", async (wos) => { const raw = wos as RawWorkOS; const response = await raw.post(`/user_management/users/${params.userId}/api_keys`, { name: params.name, @@ -552,7 +621,7 @@ const make = Effect.gen(function* () { * other key response. */ listOrgApiKeys: (organizationId: string) => - use(async (wos) => + use("organizations.listOrganizationApiKeys", async (wos) => collectWorkOSList(await wos.organizations.listOrganizationApiKeys({ organizationId })), ), @@ -564,6 +633,7 @@ const make = Effect.gen(function* () { */ createOrgApiKey: (params: { organizationId: string; name: string }) => use( + "organizations.createOrganizationApiKey", (wos) => wos.organizations.createOrganizationApiKey({ organizationId: params.organizationId, @@ -571,11 +641,12 @@ const make = Effect.gen(function* () { }) as Promise, ), - deleteApiKey: (id: string) => use((wos) => wos.apiKeys.deleteApiKey(id)), + deleteApiKey: (id: string) => + use("apiKeys.deleteApiKey", (wos) => wos.apiKeys.deleteApiKey(id)), /** List organization memberships with user details. */ listOrgMembers: (organizationId: string) => - use(async (wos) => + use("userManagement.listOrganizationMemberships", async (wos) => collectWorkOSList( await wos.userManagement.listOrganizationMemberships({ organizationId, @@ -586,7 +657,7 @@ const make = Effect.gen(function* () { /** Get a user's membership in an organization. */ getUserOrgMembership: (organizationId: string, userId: string) => - use(async (wos) => { + use("userManagement.listOrganizationMemberships", async (wos) => { const response = await wos.userManagement.listOrganizationMemberships({ organizationId, userId, @@ -596,11 +667,12 @@ const make = Effect.gen(function* () { }), /** Get a user by ID. */ - getUser: (userId: string) => use((wos) => wos.userManagement.getUser(userId)), + getUser: (userId: string) => + use("userManagement.getUser", (wos) => wos.userManagement.getUser(userId)), /** List users matching an email within one organization. */ listUsers: (params: { email: string; organizationId: string }) => - use(async (wos) => + use("userManagement.listUsers", async (wos) => collectWorkOSList( await wos.userManagement.listUsers({ email: params.email, @@ -611,7 +683,7 @@ const make = Effect.gen(function* () { /** Send an organization invitation. */ sendInvitation: (params: { email: string; organizationId: string; roleSlug?: string }) => - use((wos) => + use("userManagement.sendInvitation", (wos) => wos.userManagement.sendInvitation({ email: params.email, organizationId: params.organizationId, @@ -625,7 +697,7 @@ const make = Effect.gen(function* () { * API level, so we filter after. */ listPendingInvitations: (organizationId: string) => - use(async (wos) => + use("userManagement.listInvitations", async (wos) => collectWorkOSList( await wos.userManagement.listInvitations({ organizationId, @@ -640,7 +712,7 @@ const make = Effect.gen(function* () { /** List invitations for an email address (across all orgs). */ listInvitationsByEmail: (email: string) => - use(async (wos) => + use("userManagement.listInvitations", async (wos) => collectWorkOSList( await wos.userManagement.listInvitations({ email, @@ -650,19 +722,25 @@ const make = Effect.gen(function* () { /** Accept an invitation; returns the (now accepted) invitation. */ acceptInvitation: (invitationId: string) => - use((wos) => wos.userManagement.acceptInvitation(invitationId)), + use("userManagement.acceptInvitation", (wos) => + wos.userManagement.acceptInvitation(invitationId), + ), /** Remove an organization membership. */ deleteOrgMembership: (membershipId: string) => - use((wos) => wos.userManagement.deleteOrganizationMembership(membershipId)), + use("userManagement.deleteOrganizationMembership", (wos) => + wos.userManagement.deleteOrganizationMembership(membershipId), + ), /** Get the role for a membership. */ getOrgMembership: (membershipId: string) => - use((wos) => wos.userManagement.getOrganizationMembership(membershipId)), + use("userManagement.getOrganizationMembership", (wos) => + wos.userManagement.getOrganizationMembership(membershipId), + ), /** Update a membership's role. */ updateOrgMembershipRole: (membershipId: string, roleSlug: string) => - use((wos) => + use("userManagement.updateOrganizationMembership", (wos) => wos.userManagement.updateOrganizationMembership(membershipId, { roleSlug, }), @@ -670,15 +748,19 @@ const make = Effect.gen(function* () { /** List available roles for an organization. */ listOrgRoles: (organizationId: string) => - use((wos) => wos.organizations.listOrganizationRoles({ organizationId })), + use("organizations.listOrganizationRoles", (wos) => + wos.organizations.listOrganizationRoles({ organizationId }), + ), /** Get an organization (includes domains). */ getOrganization: (organizationId: string) => - use((wos) => wos.organizations.getOrganization(organizationId)), + use("organizations.getOrganization", (wos) => + wos.organizations.getOrganization(organizationId), + ), /** Update an organization. */ updateOrganization: (organizationId: string, name: string) => - use((wos) => + use("organizations.updateOrganization", (wos) => wos.organizations.updateOrganization({ organization: organizationId, name, @@ -690,11 +772,13 @@ const make = Effect.gen(function* () { * invitations, and domains go with it, so every member loses access. */ deleteOrganization: (organizationId: string) => - use((wos) => wos.organizations.deleteOrganization(organizationId)), + use("organizations.deleteOrganization", (wos) => + wos.organizations.deleteOrganization(organizationId), + ), /** Generate an Admin Portal link for domain verification. */ generateDomainVerificationPortalLink: (organizationId: string, returnUrl: string) => - use((wos) => + use("portal.generateLink", (wos) => wos.portal.generateLink({ organization: organizationId, intent: GeneratePortalLinkIntent.DomainVerification, @@ -704,11 +788,11 @@ const make = Effect.gen(function* () { /** Get a domain by ID. */ getOrganizationDomain: (domainId: string) => - use((wos) => wos.organizationDomains.get(domainId)), + use("organizationDomains.get", (wos) => wos.organizationDomains.get(domainId)), /** Delete a domain claim. */ deleteOrganizationDomain: (domainId: string) => - use((wos) => wos.organizationDomains.delete(domainId)), + use("organizationDomains.delete", (wos) => wos.organizationDomains.delete(domainId)), }; }); @@ -717,9 +801,10 @@ export type WorkOSClientService = Effect.Success; export class WorkOSClient extends Context.Service()( "@executor-js/cloud/WorkOSClient", ) { - static Default = Layer.effect(this)(make).pipe( - Layer.withSpan("WorkOSClient", { attributes: { module: "WorkOSClient" } }), - ); + // Deliberately unspanned: client construction is synchronous and ran per + // layer build, which produced one of the highest-volume zero-duration span + // names in the whole trace corpus while telling us nothing. + static Default = Layer.effect(this)(make); } // The boot-scoped WorkOS client root — the one neutral service the stateless diff --git a/apps/cloud/src/db/dev-db-socket-concurrency.node.test.ts b/apps/cloud/src/db/dev-db-socket-concurrency.node.test.ts index cbbec59fc..980e5edb3 100644 --- a/apps/cloud/src/db/dev-db-socket-concurrency.node.test.ts +++ b/apps/cloud/src/db/dev-db-socket-concurrency.node.test.ts @@ -21,6 +21,8 @@ // with DIFFERENT parameter counts (the exact drizzle/postgres-js shape) through // one PGLiteSocketServer and asserts zero protocol corruption. +import { setTimeout as sleep } from "node:timers/promises"; +import { connect, type Socket } from "node:net"; import { describe, expect, it } from "@effect/vitest"; import { PGlite } from "@electric-sql/pglite"; import { PGLiteSocketServer } from "@electric-sql/pglite-socket"; @@ -30,6 +32,50 @@ const PORT = 45998; const CLIENTS = 6; const QUERIES_PER_CLIENT = 40; +const makeClient = (port: number, connectTimeout = 5) => + postgres(`postgres://postgres:postgres@127.0.0.1:${port}/postgres`, { + max: 1, + idle_timeout: 0, + connect_timeout: connectTimeout, + fetch_types: false, + prepare: true, + onnotice: () => undefined, + }); + +// Hand-rolled wire client: connect and complete the trust-auth startup, so a +// test can then speak raw protocol frames (e.g. a lone Parse) that postgres.js +// would never emit on its own. Resolves after ReadyForQuery so the next write +// is its own data event — and its own queue entry — on the server. +const openWireClient = async (port: number): Promise => { + const socket: Socket = connect(port, "127.0.0.1"); + await new Promise((res, rej) => { + socket.once("connect", res); + socket.once("error", rej); + }); + const startupBody = Buffer.concat([ + Buffer.from([0, 3, 0, 0]), + Buffer.from("user\0postgres\0database\0postgres\0\0"), + ]); + const startup = Buffer.concat([Buffer.alloc(4), startupBody]); + startup.writeInt32BE(startup.length, 0); + socket.write(startup); + await new Promise((res) => { + socket.on("data", (chunk: Buffer) => { + if (chunk.includes(0x5a)) res(); // 'Z' = ReadyForQuery + }); + }); + return socket; +}; + +// A Parse frame for an unnamed statement: opens an extended-protocol pipeline +// that only a later Sync (or the server's recovery) closes. +const parseFrame = (query: string): Buffer => { + const body = Buffer.concat([Buffer.from(`\0${query}\0`), Buffer.from([0, 0])]); + const frame = Buffer.concat([Buffer.from("P"), Buffer.alloc(4), body]); + frame.writeInt32BE(4 + body.length, 1); + return frame; +}; + describe("dev-db PGlite socket under concurrent connections", () => { it( "serves interleaved multi-connection pipelines without protocol corruption", @@ -48,14 +94,7 @@ describe("dev-db PGlite socket under concurrent connections", () => { const errors: string[] = []; const worker = async (id: number) => { - const sql = postgres(`postgres://postgres:postgres@127.0.0.1:${PORT}/postgres`, { - max: 1, - idle_timeout: 0, - connect_timeout: 10, - fetch_types: false, - prepare: true, - onnotice: () => undefined, - }); + const sql = makeClient(PORT, 10); // oxlint-disable-next-line executor/no-try-catch-or-throw -- test boundary: postgres.js is promise-native and the socket must be closed on every path try { for (let q = 0; q < QUERIES_PER_CLIENT; q++) { @@ -91,4 +130,229 @@ describe("dev-db PGlite socket under concurrent connections", () => { expect(ok).toBe(CLIENTS * QUERIES_PER_CLIENT); }, ); + + // Regression for the CI e2e "cloud signIn: callback set no session (500)" + // cascade: QueryQueueManager.processQueue used to `return` out of its drain + // loop when a query REJECTED (as opposed to returning a wire-level + // ErrorResponse), leaving `processing` latched true. From then on every + // enqueue — including brand-new connections' startup packets — sat in the + // queue forever: in-flight requests hung, postgres.js reconnects died with + // CONNECT_TIMEOUT, and the whole dev stack was bricked until restart. The + // patch rejects the one entry, drops pipeline affinity, and keeps draining. + it( + "a rejected query fails one client, not the whole socket server", + { timeout: 30_000 }, + async () => { + const port = 45997; + const db = await PGlite.create(); + const server = new PGLiteSocketServer({ db, port, host: "127.0.0.1", maxConnections: 100 }); + await server.start(); + + const first = makeClient(port); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- test boundary: sockets must be closed on every path + try { + expect((await first.unsafe(`select 1 as one`))[0]).toEqual({ one: 1 }); + + // Force the NEXT protocol exchange to reject at the JS level, the shape + // PGlite produces when the shared session is broken mid-run. + const real = db.execProtocolRawStream.bind(db); + let arm = true; + (db as { execProtocolRawStream: typeof real }).execProtocolRawStream = (...args) => { + if (arm) { + arm = false; + // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- test boundary: simulating a PGlite internal failure requires a raw throw + throw new Error("synthetic PGlite failure"); + } + return real(...args); + }; + + await expect(first.unsafe(`select 2 as two`)).rejects.toThrow(); + + // The poisoned entry must take down only its own connection: a fresh + // client (new socket, full startup handshake) still gets served. + const second = makeClient(port); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- test boundary: sockets must be closed on every path + try { + expect((await second.unsafe(`select 3 as three`))[0]).toEqual({ three: 3 }); + } finally { + // oxlint-disable-next-line executor/no-promise-catch -- test boundary: a failed teardown must not mask the assertion + await second.end({ timeout: 5 }).catch(() => {}); + } + } finally { + // oxlint-disable-next-line executor/no-promise-catch -- test boundary: a failed teardown must not mask the assertion + await first.end({ timeout: 5 }).catch(() => {}); + await server.stop(); + await db.close(); + } + }, + ); + + // Regression for the sporadic `write CONNECTION_ENDED` 500s: the server's + // idleTimeout backstop used to kill ANY connection with no traffic for the + // window, which is the resting state of every healthy postgres.js pool + // connection (idle_timeout: 0) held by a long-lived scope. The backstop now + // only fires on a connection that is actually blocking the shared session — + // an open pipeline or an open transaction. + it("an idle-at-rest connection outlives the idle backstop", { timeout: 30_000 }, async () => { + const port = 45996; + const db = await PGlite.create(); + const server = new PGLiteSocketServer({ + db, + port, + host: "127.0.0.1", + maxConnections: 100, + idleTimeout: 250, + }); + await server.start(); + + const sql = makeClient(port); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- test boundary: sockets must be closed on every path + try { + expect((await sql.unsafe(`select 1 as one`))[0]).toEqual({ one: 1 }); + await sleep(900); + expect((await sql.unsafe(`select 2 as two`))[0]).toEqual({ two: 2 }); + } finally { + // oxlint-disable-next-line executor/no-promise-catch -- test boundary: a failed teardown must not mask the assertion + await sql.end({ timeout: 5 }).catch(() => {}); + await server.stop(); + await db.close(); + } + }); + + // The backstop's actual job still works: a client that opens a pipeline + // (Parse sent, never Sync) and goes silent holds queue affinity, which + // starves every other connection. The idle timer must reap exactly that + // client and hand the queue back. + it( + "a client stalled mid-pipeline is reaped and the queue recovers", + { timeout: 30_000 }, + async () => { + const port = 45995; + const db = await PGlite.create(); + const server = new PGLiteSocketServer({ + db, + port, + host: "127.0.0.1", + maxConnections: 100, + idleTimeout: 250, + }); + await server.start(); + + // Hand-rolled wire client: complete the trust-auth startup, then send a + // lone Parse. Its last frame type ('P') marks the pipeline open, so the + // handler takes affinity and every other connection queues behind it. + const staller = await openWireClient(port); + staller.write(parseFrame("select 1")); + + const bystander = makeClient(port, 10); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- test boundary: sockets must be closed on every path + try { + // Connects and queries only once the staller is reaped (~250ms). + expect((await bystander.unsafe(`select 4 as four`))[0]).toEqual({ four: 4 }); + } finally { + // oxlint-disable-next-line executor/no-promise-catch -- test boundary: a failed teardown must not mask the assertion + await bystander.end({ timeout: 5 }).catch(() => {}); + staller.destroy(); + await server.stop(); + await db.close(); + } + }, + ); + + // Regression for the reap SLOT LEAK: detach(true) removes the socket's + // listeners before destroying it, so a server-initiated teardown (the idle + // backstop) never fired the server's 'close' bookkeeping — the reaped + // handler stayed in the server's handlers set forever, burning one + // maxConnections slot per reap. Enough reaps over a long run and the server + // answers every NEW connection with "Too many connections" while the + // process, the port, and PGlite are all healthy — postgres.js surfaces that + // as the same CONNECT_TIMEOUT cascade as the queue wedges. The server now + // drops the handler when it dispatches its terminal error. + it("reaped handlers release their connection slots", { timeout: 30_000 }, async () => { + const port = 45993; + const db = await PGlite.create(); + const server = new PGLiteSocketServer({ + db, + port, + host: "127.0.0.1", + maxConnections: 2, + idleTimeout: 250, + }); + await server.start(); + + // oxlint-disable-next-line executor/no-try-catch-or-throw -- test boundary: sockets must be closed on every path + try { + // Burn through more reaps than there are slots: each staller opens a + // pipeline and goes silent, so the idle backstop reaps it (the server + // destroys the socket — its 'close' marks that reap complete). + for (let i = 0; i < 3; i++) { + const staller = await openWireClient(port); + staller.write(parseFrame(`select ${i + 1}`)); + await new Promise((res) => staller.once("close", res)); + } + + expect( + server.getStats().activeConnections, + "reaped handlers stay counted against maxConnections", + ).toBe(0); + + const sql = makeClient(port); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- test boundary: sockets must be closed on every path + try { + expect((await sql.unsafe(`select 6 as six`))[0]).toEqual({ six: 6 }); + } finally { + // oxlint-disable-next-line executor/no-promise-catch -- test boundary: a failed teardown must not mask the assertion + await sql.end({ timeout: 5 }).catch(() => {}); + } + } finally { + await server.stop(); + await db.close(); + } + }); + + // Regression for the second wedge mode behind the same CI cascade: a client + // whose socket dies WHILE its pipeline-opening entry is executing. detach() + // clears pipeline affinity before the entry finishes, so the queue then + // assigned affinity to the already-dead handler — and nothing ever cleared + // it: the dead handler has no timers left, and every other connection + // (including fresh startups) queued behind the ghost forever. The queue now + // tracks detached handlers and repairs affinity they can no longer release. + it( + "a client that dies mid-execution does not leave the queue pinned to its ghost", + { timeout: 30_000 }, + async () => { + const port = 45994; + const db = await PGlite.create(); + + // Hold the marker query in flight long enough that the disconnect below + // reliably lands while the entry is EXECUTING (after detach's cleanup, + // before the queue takes affinity for it). + const real = db.execProtocolRawStream.bind(db); + (db as { execProtocolRawStream: typeof real }).execProtocolRawStream = async (...args) => { + if (Buffer.from(args[0]).includes("ghost_marker")) await sleep(300); + return real(...args); + }; + + const server = new PGLiteSocketServer({ db, port, host: "127.0.0.1", maxConnections: 100 }); + await server.start(); + + const ghost = await openWireClient(port); + ghost.write(parseFrame("select 'ghost_marker'")); + // Give the data event time to reach the queue and start executing, then + // die without a trace mid-flight. + await sleep(100); + ghost.destroy(); + + const bystander = makeClient(port); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- test boundary: sockets must be closed on every path + try { + expect((await bystander.unsafe(`select 5 as five`))[0]).toEqual({ five: 5 }); + } finally { + // oxlint-disable-next-line executor/no-promise-catch -- test boundary: a failed teardown must not mask the assertion + await bystander.end({ timeout: 5 }).catch(() => {}); + await server.stop(); + await db.close(); + } + }, + ); }); diff --git a/apps/cloud/src/edge/docs.ts b/apps/cloud/src/edge/docs.ts index d9f18433f..fabf3773e 100644 --- a/apps/cloud/src/edge/docs.ts +++ b/apps/cloud/src/edge/docs.ts @@ -6,43 +6,23 @@ // base path, so the pathname is forwarded UNCHANGED — only the host/proto swap // to the upstream origin (unlike the PostHog proxy, which strips its prefix). // -// Like the PostHog/Sentry tunnels (and unlike the marketing proxy, which needs -// the prod-only `env.MARKETING` service binding), this is a plain external -// `fetch`, so it runs on every host — `/docs` previews against live Mintlify in -// local dev too. `/docs` is distinct from the app-owned `/api/docs` (Swagger), -// so this never shadows an Effect-served route. +// The matching, upstream construction, and client span live in `./passthrough`, +// which server.ts dispatches BEFORE Start loads: forwarding a docs page must +// not pay for the whole server graph. This middleware stays registered so hosts +// that reach Start by another entry (local dev) keep identical behavior; in the +// deployed Worker it is unreachable. `/docs` is distinct from the app-owned +// `/api/docs` (Swagger), so this never shadows an Effect-served route. // --------------------------------------------------------------------------- import { createMiddleware } from "@tanstack/react-start"; -const DOCS_UPSTREAM_HOST = "executor.mintlify.dev"; +import { docsProxyResponse, isDocsPath } from "./passthrough"; -export const isDocsPath = (pathname: string) => - pathname === "/docs" || pathname.startsWith("/docs/"); - -// Build the upstream request for an already-classified `/docs` path. Caller -// guarantees `isDocsPath(pathname)` — we only swap the origin and fix up the -// forwarding headers, preserving method, body, path, and query. -export const buildDocsUpstream = (request: Request): Request => { - const url = new URL(request.url); - const forwardedHost = url.host; - - url.hostname = DOCS_UPSTREAM_HOST; - url.protocol = "https:"; - url.port = ""; - - const upstream = new Request(url, request); - // Mintlify keys canonical links off the public host; tell it the real one. - upstream.headers.set("X-Forwarded-Host", forwardedHost); - upstream.headers.set("X-Forwarded-Proto", "https"); - // Never leak the executor.sh session cookie to the docs origin. - upstream.headers.delete("cookie"); - return upstream; -}; +export { buildDocsUpstream, isDocsPath } from "./passthrough"; export const docsProxyMiddleware = createMiddleware({ type: "request" }).server( ({ pathname, request, next }) => { if (!isDocsPath(pathname)) return next(); - return fetch(buildDocsUpstream(request)); + return docsProxyResponse(request, pathname); }, ); diff --git a/apps/cloud/src/edge/index.ts b/apps/cloud/src/edge/index.ts index 1b1cbe9ad..6b58c2eda 100644 --- a/apps/cloud/src/edge/index.ts +++ b/apps/cloud/src/edge/index.ts @@ -1,11 +1,10 @@ // --------------------------------------------------------------------------- -// Edge concerns — the analytics/marketing/docs request middlewares that run at -// the worker edge BEFORE the app's own mcp + api dispatch. None of these touch -// the Effect app layer; they proxy or tunnel to external services (the -// marketing worker, Sentry, PostHog, Mintlify docs). +// Edge concerns — request middleware that runs before the app's own mcp + api +// dispatch. These proxy or tunnel to external services without touching the +// Effect app layer. Marketing is dispatched even earlier, from server.ts, so a +// public page never loads the TanStack Start graph. // --------------------------------------------------------------------------- -export { marketingMiddleware } from "./marketing"; export { sentryTunnelMiddleware } from "./sentry-tunnel"; export { posthogProxyMiddleware } from "./posthog"; export { docsProxyMiddleware } from "./docs"; diff --git a/apps/cloud/src/edge/marketing.test.ts b/apps/cloud/src/edge/marketing.test.ts index 38019ce9b..f96958a0a 100644 --- a/apps/cloud/src/edge/marketing.test.ts +++ b/apps/cloud/src/edge/marketing.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "@effect/vitest"; -import { isMarketingPath } from "./marketing"; +import { isMarketingPath, marketingProxyRequest } from "./marketing"; // On executor.sh the marketing middleware proxies an allow-list of paths to the // `executor-marketing` worker; everything else falls through to the auth-gated @@ -12,6 +12,9 @@ describe("isMarketingPath", () => { "/home", "/privacy", "/terms", + "/about-executor", + "/google-oauth", + "/google-workspace", "/blog", "/blog/", "/blog/some-post", @@ -37,3 +40,54 @@ describe("isMarketingPath", () => { }); } }); + +describe("marketingProxyRequest", () => { + it("routes a signed-out homepage request", () => { + const request = new Request("https://executor.sh/?source=test"); + + const proxied = marketingProxyRequest(request); + + expect(proxied?.url).toBe("https://executor.sh/?source=test"); + }); + + it("leaves the signed-in homepage with the cloud application", () => { + const request = new Request("https://executor.sh/", { + headers: { cookie: "other=value; wos-session=sealed" }, + }); + + expect(marketingProxyRequest(request)).toBeNull(); + }); + + it("routes public content even when a session cookie is present", () => { + const request = new Request("https://executor.sh/blog/post", { + headers: { cookie: "wos-session=sealed" }, + }); + + expect(marketingProxyRequest(request)?.url).toBe("https://executor.sh/blog/post"); + }); + + it("rewrites the public home alias to the marketing root", () => { + const request = new Request("https://executor.sh/home?source=test"); + + expect(marketingProxyRequest(request)?.url).toBe("https://executor.sh/?source=test"); + }); + + it("preserves the request method, headers, and body", async () => { + const request = new Request("https://executor.sh/_astro/_ph/capture", { + method: "POST", + headers: { "content-type": "application/json", "x-request-id": "request-1" }, + body: JSON.stringify({ event: "test" }), + }); + + const proxied = marketingProxyRequest(request); + + expect(proxied?.method).toBe("POST"); + expect(proxied?.headers.get("x-request-id")).toBe("request-1"); + await expect(proxied?.json()).resolves.toEqual({ event: "test" }); + }); + + it("does not proxy non-production hosts or app-owned paths", () => { + expect(marketingProxyRequest(new Request("http://executor-cloud.localhost/"))).toBeNull(); + expect(marketingProxyRequest(new Request("https://executor.sh/login"))).toBeNull(); + }); +}); diff --git a/apps/cloud/src/edge/marketing.ts b/apps/cloud/src/edge/marketing.ts index ee473ce68..b51ef0b45 100644 --- a/apps/cloud/src/edge/marketing.ts +++ b/apps/cloud/src/edge/marketing.ts @@ -3,14 +3,10 @@ // // On the production domain (`executor.sh`), marketing paths and the // unauthenticated landing page are served by the separate `executor-marketing` -// worker (bound as `env.MARKETING`). In local dev that worker isn't running, so -// unauthenticated visits fall through to the cloud app's routes (the sign-in -// page). +// worker. This module deliberately has no TanStack Start or cloud application +// imports: the Worker entry calls it before loading the Start server graph. // --------------------------------------------------------------------------- -import { env } from "cloudflare:workers"; -import { createMiddleware } from "@tanstack/react-start"; - import { parseCookie } from "../auth/cookies"; const MARKETING_PATHS = [ @@ -18,6 +14,9 @@ const MARKETING_PATHS = [ "/setup", "/privacy", "/terms", + "/about-executor", + "/google-oauth", + "/google-workspace", "/blog", "/llms.txt", "/api/detect", @@ -27,33 +26,25 @@ const MARKETING_PATHS = [ "/pattern-graph-paper.svg", ]; -export const isMarketingPath = (pathname: string) => - MARKETING_PATHS.some((p) => pathname === p || pathname.startsWith(`${p}/`)); - -const getMarketingWorker = () => env.MARKETING as { fetch: typeof fetch } | undefined; - -export const marketingMiddleware = createMiddleware({ type: "request" }).server( - async ({ pathname, request, next }) => { - // Only proxy to the marketing worker on the production domain. In local - // dev we don't run `executor-marketing`, so unauthenticated visits fall - // through to the cloud app's routes (which show the sign-in page). - const host = new URL(request.url).hostname; - if (host !== "executor.sh") return next(); +const SESSION_COOKIE = "wos-session"; - const shouldProxyToMarketing = - isMarketingPath(pathname) || - (pathname === "/" && !parseCookie(request.headers.get("cookie"), "wos-session")); - - if (!shouldProxyToMarketing) return next(); - - const marketing = getMarketingWorker(); - if (!marketing) return next(); +/** Whether an exact pathname belongs to the public marketing worker. */ +export const isMarketingPath = (pathname: string): boolean => + MARKETING_PATHS.some((p) => pathname === p || pathname.startsWith(`${p}/`)); - const url = new URL(request.url); - // Rewrite /home to / so marketing worker serves its homepage - if (pathname === "/home") { - url.pathname = "/"; - } - return marketing.fetch(new Request(url, request)); - }, -); +/** + * Project a production request onto the marketing service-binding request. + * Returns `null` when the cloud application owns the request instead. + */ +export const marketingProxyRequest = (request: Request): Request | null => { + const url = new URL(request.url); + if (url.hostname !== "executor.sh") return null; + + const shouldProxy = + isMarketingPath(url.pathname) || + (url.pathname === "/" && !parseCookie(request.headers.get("cookie"), SESSION_COOKIE)); + if (!shouldProxy) return null; + + if (url.pathname === "/home") url.pathname = "/"; + return new Request(url, request); +}; diff --git a/apps/cloud/src/edge/passthrough.test.ts b/apps/cloud/src/edge/passthrough.test.ts new file mode 100644 index 000000000..2f5bbd35a --- /dev/null +++ b/apps/cloud/src/edge/passthrough.test.ts @@ -0,0 +1,85 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "@effect/vitest"; + +import { + buildDocsUpstream, + buildPosthogUpstream, + isDocsPath, + isPosthogPath, + passthroughResponse, + POSTHOG_PROXY_PATH, +} from "./passthrough"; + +describe("passthrough matching", () => { + it("claims /docs and everything under it, but not /api/docs", () => { + expect(isDocsPath("/docs")).toBe(true); + expect(isDocsPath("/docs/concepts/policies")).toBe(true); + // The app-owned Swagger route must keep reaching the Effect app. + expect(isDocsPath("/api/docs")).toBe(false); + expect(isDocsPath("/docsearch")).toBe(false); + }); + + it("claims the PostHog proxy path and its subtree only", () => { + expect(isPosthogPath(POSTHOG_PROXY_PATH)).toBe(true); + expect(isPosthogPath(`${POSTHOG_PROXY_PATH}/i/v0/e/`)).toBe(true); + expect(isPosthogPath(`${POSTHOG_PROXY_PATH}extra`)).toBe(false); + expect(isPosthogPath("/api/connections")).toBe(false); + }); + + it("returns null for app paths so they fall through to normal dispatch", () => { + expect(passthroughResponse(new Request("https://executor.sh/"), "/")).toBeNull(); + expect( + passthroughResponse(new Request("https://executor.sh/api/connections"), "/api/connections"), + ).toBeNull(); + expect(passthroughResponse(new Request("https://executor.sh/mcp"), "/mcp")).toBeNull(); + }); +}); + +describe("upstream construction", () => { + it("forwards the docs path unchanged and strips the session cookie", () => { + const upstream = buildDocsUpstream( + new Request("https://executor.sh/docs/concepts/policies?x=1", { + headers: { cookie: "wos-session=secret" }, + }), + ); + const url = new URL(upstream.url); + expect(url.hostname).toBe("executor.mintlify.dev"); + expect(url.pathname).toBe("/docs/concepts/policies"); + expect(url.search).toBe("?x=1"); + expect(upstream.headers.get("X-Forwarded-Host")).toBe("executor.sh"); + expect(upstream.headers.get("cookie")).toBeNull(); + }); + + it("strips the proxy prefix for PostHog and splits ingest from assets", () => { + const ingest = buildPosthogUpstream( + new Request(`https://executor.sh${POSTHOG_PROXY_PATH}/i/v0/e/`), + `${POSTHOG_PROXY_PATH}/i/v0/e/`, + ); + expect(new URL(ingest.url).hostname).toBe("us.i.posthog.com"); + expect(new URL(ingest.url).pathname).toBe("/i/v0/e/"); + + const assets = buildPosthogUpstream( + new Request(`https://executor.sh${POSTHOG_PROXY_PATH}/static/array.js`), + `${POSTHOG_PROXY_PATH}/static/array.js`, + ); + expect(new URL(assets.url).hostname).toBe("us-assets.i.posthog.com"); + expect(new URL(assets.url).pathname).toBe("/static/array.js"); + }); +}); + +describe("Start-graph independence", () => { + // The entire point of this module is that server.ts can answer a proxy + // request WITHOUT importing TanStack Start. An import of the app or of + // `@tanstack/react-start` here silently reintroduces the ~3.1s cold-isolate + // `loadEntries` cost this module exists to avoid, and nothing else would + // catch it — the behavior stays correct, only slow. + it("imports neither TanStack Start nor an app module", () => { + const source = readFileSync(new URL("./passthrough.ts", import.meta.url), "utf8"); + const imports = [...source.matchAll(/^\s*import[^"']*["']([^"']+)["']/gm)].map((m) => m[1]); + expect(imports.length).toBeGreaterThan(0); + for (const specifier of imports) { + expect(specifier).not.toMatch(/@tanstack/); + expect(specifier).not.toMatch(/^\.\.\//); + } + }); +}); diff --git a/apps/cloud/src/edge/passthrough.ts b/apps/cloud/src/edge/passthrough.ts new file mode 100644 index 000000000..76049436c --- /dev/null +++ b/apps/cloud/src/edge/passthrough.ts @@ -0,0 +1,138 @@ +// --------------------------------------------------------------------------- +// Pure passthrough proxies — dispatched from the Worker entry, before Start. +// --------------------------------------------------------------------------- +// +// `/docs` and the PostHog proxy forward to an external origin and never touch +// the router, React, or the Effect app. They lived in Start's request +// middleware, which meant each one still paid Start's lazy `loadEntries` +// import of the whole server graph before it could forward a request. +// +// Measured on production (2026-08-18), splitting the two costs apart on a cold +// isolate: the graph import is p50 **3.1s** while the request's own work is +// p50 **33ms**. And essentially every request is cold — `worker.dispatch` ran +// 1,666 requests across 1,608 isolates (1.04 req/isolate), because `/mcp` +// dispatches before `fetchHandler` and so never warms the graph. A `/docs` +// page took 3-6s through the Worker against 0.098s straight from the upstream. +// +// So these move to the Worker entry, exactly as marketing did: classify and +// forward before anything imports Start. This module therefore must NOT import +// `@tanstack/react-start` or any app module — that import is the cost it +// exists to avoid. +// +// The middleware wrappers in `./docs` and `./posthog` stay registered. In the +// deployed Worker they become unreachable (server.ts answers first), but they +// keep the behavior identical on any host that reaches Start by another entry +// (local dev), and they source their matching from here so the two can't drift. +// --------------------------------------------------------------------------- + +import { SpanKind, SpanStatusCode, trace } from "@opentelemetry/api"; + +const DOCS_UPSTREAM_HOST = "executor.mintlify.dev"; +/** PostHog's US ingest origin. Exported because the server-side feature-flag + * gate (`../analytics/ema-rollout`) calls the same origin directly, and the + * two must not be able to drift onto different PostHog regions. */ +export const POSTHOG_INGEST_HOST = "us.i.posthog.com"; +const POSTHOG_ASSETS_HOST = "us-assets.i.posthog.com"; + +export const POSTHOG_PROXY_PATH = `/api/${( + import.meta.env.VITE_PUBLIC_ANALYTICS_PATH ?? "a" +).replace(/^\/+|\/+$/g, "")}`; + +// The proxy fetch gets its own client span: `/docs` requests otherwise render +// as a single opaque server span, and during the Aug 2026 regression there +// was no way to tell upstream (Mintlify/Vercel) latency from worker-side +// dispatch cost. The noop tracer applies when no provider is installed +// (local dev without AXIOM_TOKEN), so this is free there. +const tracer = trace.getTracer("executor-cloud-docs-proxy"); + +export const isDocsPath = (pathname: string): boolean => + pathname === "/docs" || pathname.startsWith("/docs/"); + +export const isPosthogPath = (pathname: string): boolean => + pathname === POSTHOG_PROXY_PATH || pathname.startsWith(`${POSTHOG_PROXY_PATH}/`); + +/** + * Build the upstream request for an already-classified `/docs` path. Caller + * guarantees `isDocsPath(pathname)` — we only swap the origin and fix up the + * forwarding headers, preserving method, body, path, and query. + */ +export const buildDocsUpstream = (request: Request): Request => { + const url = new URL(request.url); + const forwardedHost = url.host; + + url.hostname = DOCS_UPSTREAM_HOST; + url.protocol = "https:"; + url.port = ""; + + const upstream = new Request(url, request); + // Mintlify keys canonical links off the public host; tell it the real one. + upstream.headers.set("X-Forwarded-Host", forwardedHost); + upstream.headers.set("X-Forwarded-Proto", "https"); + // Never leak the executor.sh session cookie to the docs origin. + upstream.headers.delete("cookie"); + return upstream; +}; + +/** Build the upstream request for an already-classified PostHog proxy path. */ +export const buildPosthogUpstream = (request: Request, pathname: string): Request => { + const url = new URL(request.url); + url.hostname = pathname.startsWith(`${POSTHOG_PROXY_PATH}/static/`) + ? POSTHOG_ASSETS_HOST + : POSTHOG_INGEST_HOST; + url.protocol = "https:"; + url.port = ""; + url.pathname = pathname.slice(POSTHOG_PROXY_PATH.length) || "/"; + + const upstream = new Request(url, request); + upstream.headers.delete("cookie"); + return upstream; +}; + +export const docsProxyResponse = (request: Request, pathname: string): Promise => + tracer.startActiveSpan( + `http.client ${request.method}`, + { + kind: SpanKind.CLIENT, + attributes: { + "server.address": DOCS_UPSTREAM_HOST, + "url.path": pathname, + "http.request.method": request.method, + }, + }, + async (span) => { + // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; observe upstream response/error for span status, then pass both through unchanged + try { + const response = await fetch(buildDocsUpstream(request)); + span.setAttribute("http.response.status_code", response.status); + if (response.status >= 500) { + span.setStatus({ code: SpanStatusCode.ERROR, message: `HTTP ${response.status}` }); + } + return response; + } catch (err) { + // oxlint-disable-next-line executor/no-instanceof-error, executor/no-unknown-error-message -- adapter boundary: fetch rejects untyped; normalized only for the OTel span record, the original error is rethrown below + const cause = err instanceof Error ? err : String(err); + span.recordException(cause); + // oxlint-disable-next-line executor/no-unknown-error-message -- adapter boundary: same normalization as the recordException line above + const message = typeof cause === "string" ? cause : cause.message; + span.setStatus({ code: SpanStatusCode.ERROR, message }); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; preserve the original rejection for the platform handler + throw err; + } finally { + span.end(); + } + }, + ); + +/** + * Answer a pure passthrough request without loading the Start server graph. + * Returns `null` when the request belongs to the app, so the caller falls + * through to normal dispatch. + */ +export const passthroughResponse = ( + request: Request, + pathname: string, +): Promise | null => { + if (isDocsPath(pathname)) return docsProxyResponse(request, pathname); + if (isPosthogPath(pathname)) return fetch(buildPosthogUpstream(request, pathname)); + return null; +}; diff --git a/apps/cloud/src/edge/posthog.ts b/apps/cloud/src/edge/posthog.ts index 6badd574f..a9a035b4d 100644 --- a/apps/cloud/src/edge/posthog.ts +++ b/apps/cloud/src/edge/posthog.ts @@ -3,33 +3,20 @@ // first-party path and we forward to PostHog's ingest + asset hosts. Keeps // events flowing past adblockers that match *.posthog.com. See // https://posthog.com/docs/advanced/proxy/cloudflare +// +// The matching and forwarding live in `./passthrough`, which server.ts +// dispatches BEFORE Start loads (a proxy must not pay for the server graph). +// This middleware stays registered so hosts that reach Start by another entry +// keep identical behavior; in the deployed Worker it is unreachable. // --------------------------------------------------------------------------- import { createMiddleware } from "@tanstack/react-start"; -const POSTHOG_INGEST_HOST = "us.i.posthog.com"; -const POSTHOG_ASSETS_HOST = "us-assets.i.posthog.com"; -const POSTHOG_PROXY_PATH = `/api/${(import.meta.env.VITE_PUBLIC_ANALYTICS_PATH ?? "a").replace( - /^\/+|\/+$/g, - "", -)}`; +import { buildPosthogUpstream, isPosthogPath } from "./passthrough"; export const posthogProxyMiddleware = createMiddleware({ type: "request" }).server( ({ pathname, request, next }) => { - if (pathname !== POSTHOG_PROXY_PATH && !pathname.startsWith(`${POSTHOG_PROXY_PATH}/`)) { - return next(); - } - - const url = new URL(request.url); - url.hostname = pathname.startsWith(`${POSTHOG_PROXY_PATH}/static/`) - ? POSTHOG_ASSETS_HOST - : POSTHOG_INGEST_HOST; - url.protocol = "https:"; - url.port = ""; - url.pathname = pathname.slice(POSTHOG_PROXY_PATH.length) || "/"; - - const upstream = new Request(url, request); - upstream.headers.delete("cookie"); - return fetch(upstream); + if (!isPosthogPath(pathname)) return next(); + return fetch(buildPosthogUpstream(request, pathname)); }, ); diff --git a/apps/cloud/src/engine/execution-rate-limit.node.test.ts b/apps/cloud/src/engine/execution-rate-limit.node.test.ts new file mode 100644 index 000000000..678697edc --- /dev/null +++ b/apps/cloud/src/engine/execution-rate-limit.node.test.ts @@ -0,0 +1,433 @@ +import { afterEach, describe, expect, it } from "@effect/vitest"; +import { env } from "cloudflare:workers"; +import { Data, Effect } from "effect"; +import type * as Tracer from "effect/Tracer"; + +import type { ExecutionEngine } from "@executor-js/execution"; + +import { + ExecutionRateLimiterDO, + makeCloudExecutionRateLimiter, + makeExecutionRateLimiter, + RateLimitCounterError, +} from "./execution-rate-limit"; +import { RATE_LIMIT_BLOCKED_MESSAGE } from "./execution-limit-messages"; + +const ORG = "org_test"; + +/** Stands in for whatever the real lookups fail with (Autumn down, DO down). */ +class UpstreamDownError extends Data.TaggedError("UpstreamDownError")<{ + readonly which: string; +}> {} + +// A stand-in engine: `execute` resolves to a marker, so a test tells an allowed +// execution (marker) from a blocked one (the gate's error result) by which of +// the two came back. A blocked decision never reaches the engine at all. +const engineStub: ExecutionEngine = { + execute: () => Effect.succeed({ result: "ran" }), + executeWithPause: () => Effect.succeed({ status: "completed", result: { result: "ran" } }), + resume: () => Effect.succeed(null), + getPausedExecution: () => Effect.succeed(null), + pausedExecutionCount: () => Effect.succeed(0), + hasPausedExecutions: () => Effect.succeed(false), + getDescription: Effect.succeed("stub"), +}; + +/** Counter that hands out a caller-controlled sequence of counts. */ +const countingIncrement = (counts: ReadonlyArray) => { + let calls = 0; + return () => Effect.succeed(counts[Math.min(calls++, counts.length - 1)] ?? 0); +}; + +const runExecute = (limiter: ReturnType) => + Effect.runPromise( + limiter.decorate(ORG, engineStub).execute("code", { + // Never invoked: the stub ignores it, and a blocked execution never runs. + onElicitation: () => Effect.die("elicitation is not exercised here"), + }), + ); + +describe("execution rate limiter — paid exemption", () => { + it("allows executions under the cap without consulting the exemption", async () => { + let exemptionCalls = 0; + const limiter = makeExecutionRateLimiter(countingIncrement([1]), { + limit: 10, + isExempt: () => { + exemptionCalls += 1; + return Effect.succeed(false); + }, + }); + + expect(await runExecute(limiter)).toMatchObject({ result: "ran" }); + // The whole point of resolving lazily: the common path costs no lookup. + expect(exemptionCalls).toBe(0); + }); + + it("blocks a non-exempt org over the cap", async () => { + const limiter = makeExecutionRateLimiter(countingIncrement([11]), { + limit: 10, + isExempt: () => Effect.succeed(false), + }); + + expect(await runExecute(limiter)).toMatchObject({ + result: null, + error: RATE_LIMIT_BLOCKED_MESSAGE, + }); + }); + + it("allows an exempt org over the cap", async () => { + const limiter = makeExecutionRateLimiter(countingIncrement([11]), { + limit: 10, + isExempt: () => Effect.succeed(true), + }); + + expect(await runExecute(limiter)).toMatchObject({ result: "ran" }); + }); + + it("caches the exemption so a paid org past the cap looks it up once", async () => { + let exemptionCalls = 0; + const limiter = makeExecutionRateLimiter(countingIncrement([11, 12, 13]), { + limit: 10, + exemptionTtlMs: 60_000, + now: () => 1_000, + isExempt: () => { + exemptionCalls += 1; + return Effect.succeed(true); + }, + }); + + for (let i = 0; i < 3; i += 1) + expect(await runExecute(limiter)).toMatchObject({ result: "ran" }); + expect(exemptionCalls).toBe(1); + }); + + it("re-resolves once the cached exemption expires", async () => { + let exemptionCalls = 0; + let clock = 1_000; + const limiter = makeExecutionRateLimiter(countingIncrement([11, 12]), { + limit: 10, + exemptionTtlMs: 1_000, + now: () => clock, + isExempt: () => { + exemptionCalls += 1; + return Effect.succeed(true); + }, + }); + + await runExecute(limiter); + clock += 5_000; + await runExecute(limiter); + expect(exemptionCalls).toBe(2); + }); + + it("blocks when the exemption cannot be resolved and nothing is cached", async () => { + // Deliberately NOT fail-open: an unresolvable exemption during an Autumn + // outage must not switch the backstop off, which is the one scenario it + // exists to cover. + const limiter = makeExecutionRateLimiter(countingIncrement([11]), { + limit: 10, + isExempt: () => Effect.fail(new UpstreamDownError({ which: "autumn" })), + }); + + expect(await runExecute(limiter)).toMatchObject({ + result: null, + error: RATE_LIMIT_BLOCKED_MESSAGE, + }); + }); + + it("honours a stale exemption when a later lookup fails", async () => { + let clock = 1_000; + let shouldFail = false; + const limiter = makeExecutionRateLimiter(countingIncrement([11, 12]), { + limit: 10, + exemptionTtlMs: 1_000, + now: () => clock, + isExempt: () => + shouldFail ? Effect.fail(new UpstreamDownError({ which: "autumn" })) : Effect.succeed(true), + }); + + expect(await runExecute(limiter)).toMatchObject({ result: "ran" }); + + // Cache expires and Autumn is now unreachable: the known-paid org keeps + // running rather than getting blocked mid-workload by a blip. + clock += 5_000; + shouldFail = true; + expect(await runExecute(limiter)).toMatchObject({ result: "ran" }); + }); + + it("fails open when the counter itself is unreachable", async () => { + const limiter = makeExecutionRateLimiter( + (organizationId) => + Effect.fail( + new RateLimitCounterError({ + organizationId, + code: "unknown", + reason: "counter DO unreachable", + cause: null, + }), + ), + { + limit: 10, + isExempt: () => Effect.succeed(false), + }, + ); + + expect(await runExecute(limiter)).toMatchObject({ result: "ran" }); + }); + + it("applies the cap when no exemption predicate is wired", async () => { + const limiter = makeExecutionRateLimiter(countingIncrement([11]), { limit: 10 }); + + expect(await runExecute(limiter)).toMatchObject({ + result: null, + error: RATE_LIMIT_BLOCKED_MESSAGE, + }); + }); +}); + +// --------------------------------------------------------------------------- +// Counter observability — the production DO wiring. +// +// The counter increment used to be a bare one-argument `Effect.tryPromise` +// with no span, so a Durable Object fault reached error reporting as +// `UnknownError: An error occurred in Effect.tryPromise` with no application +// frames, and the 2s check budget was invisible in traces. These tests pin +// the named spans, the typed/classified failure, and the timeout override — +// all read through the REAL `makeCloudExecutionRateLimiter` wiring against a +// fake `EXECUTION_RATE_LIMITER` binding. +// --------------------------------------------------------------------------- + +type RecordedSpan = { + readonly name: string; + readonly attributes: Map; +}; + +/** A tracer that keeps every span it is asked to open, with its attributes. */ +const recordingTracer = (recorded: Array): Tracer.Tracer => { + let nextId = 1; + return { + span: (options) => { + let status: Tracer.SpanStatus = { _tag: "Started", startTime: options.startTime }; + const attributes = new Map(); + recorded.push({ name: options.name, attributes }); + const id = String(nextId++).padStart(16, "0"); + return { + _tag: "Span", + name: options.name, + spanId: id, + traceId: "00000000000000000000000000000001", + parent: options.parent, + annotations: options.annotations, + get status() { + return status; + }, + attributes, + links: options.links, + sampled: options.sampled, + kind: options.kind, + end: (endTime, exit) => { + status = { _tag: "Ended", startTime: options.startTime, endTime, exit }; + }, + attribute: (key, value) => { + attributes.set(key, value); + }, + event: () => undefined, + addLinks: () => undefined, + }; + }, + }; +}; + +const spanNamed = (recorded: ReadonlyArray, name: string): RecordedSpan => { + const span = recorded.find((candidate) => candidate.name === name); + expect( + span, + `a span named ${name} is recorded (got: ${recorded.map((s) => s.name).join(", ") || "none"})`, + ).toBeDefined(); + return span ?? { name, attributes: new Map() }; +}; + +/** A counter-DO namespace whose single RPC method behaves as the test says. */ +const namespaceReturning = (increment: () => Promise) => ({ + idFromName: (name: string) => ({ toString: () => name }), + get: () => ({ increment }), +}); + +/** The three worker vars the production limiter reads at construction. */ +type CounterEnv = { + EXECUTION_RATE_LIMITER?: unknown; + EXECUTION_RATE_LIMIT_PER_HOUR?: string; + EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS?: string; +}; + +const cloudEnv: CounterEnv = env; +const savedEnv: CounterEnv = { ...cloudEnv }; + +// Restore rather than leak: the limiter is built from the worker env, and a +// stale binding or budget would silently retune a later test. +afterEach(() => { + delete cloudEnv.EXECUTION_RATE_LIMITER; + delete cloudEnv.EXECUTION_RATE_LIMIT_PER_HOUR; + delete cloudEnv.EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS; + Object.assign(cloudEnv, savedEnv); +}); + +// The exact platform fault behind the production issue: Cloudflare resets the +// object and the RPC rejects with a plain Error. The reference id is synthetic. +const DO_STORAGE_RESET = + "Internal error in Durable Object storage caused object to be reset; reference = 0000000000000000"; + +// oxlint-disable-next-line executor/no-promise-reject, executor/no-error-constructor -- boundary: the Durable Object RPC is a promise that rejects with a platform Error; the test double has to fail the same way for the classification to mean anything +const rejectStorageReset = (): Promise => Promise.reject(new Error(DO_STORAGE_RESET)); + +/** Build the production limiter against a fake binding and worker env. */ +const cloudLimiter = (options: { + readonly namespace: ReturnType; + readonly limit: string; + readonly timeoutMs?: string; +}): ReturnType => { + cloudEnv.EXECUTION_RATE_LIMITER = options.namespace; + cloudEnv.EXECUTION_RATE_LIMIT_PER_HOUR = options.limit; + if (options.timeoutMs !== undefined) + cloudEnv.EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS = options.timeoutMs; + return makeCloudExecutionRateLimiter(() => Effect.succeed(false)); +}; + +const runExecuteTraced = ( + limiter: ReturnType, + recorded: Array, +) => + Effect.runPromise( + limiter + .decorate(ORG, engineStub) + .execute("code", { onElicitation: () => Effect.die("elicitation is not exercised here") }) + .pipe(Effect.withTracer(recordingTracer(recorded))), + ); + +describe("execution rate limiter — counter observability", () => { + it("reports a counter-DO fault as a typed, classified error on a named span", async () => { + const recorded: Array = []; + const limiter = cloudLimiter({ + namespace: namespaceReturning(rejectStorageReset), + limit: "10", + }); + const result = await runExecuteTraced(limiter, recorded); + + // Fail-open semantics are unchanged: a broken counter never blocks. + expect(result).toMatchObject({ result: "ran" }); + + const increment = spanNamed(recorded, "rate_limit.increment"); + expect(increment.attributes.get("rate_limit.counter.error_tag")).toBe("RateLimitCounterError"); + expect(increment.attributes.get("rate_limit.counter.error_code")).toBe("storage_reset"); + + const check = spanNamed(recorded, "rate_limit.check"); + expect(check.attributes.get("rate_limit.check.failed_open")).toBe(true); + expect(check.attributes.get("rate_limit.check.timed_out")).toBe(false); + expect(check.attributes.get("rate_limit.check.error_tag")).toBe("RateLimitCounterError"); + }); + + it("honours the EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS override", async () => { + const recorded: Array = []; + const limiter = cloudLimiter({ + // Answers well past the tiny budget, with a count far over the cap. If + // the override were ignored the default 2s budget would let that count + // through and BLOCK the execution; honouring it times the check out and + // fails open instead, so the two outcomes are distinguishable without + // measuring wall time. + namespace: namespaceReturning( + () => new Promise((resolve) => setTimeout(() => resolve(999), 200)), + ), + limit: "10", + timeoutMs: "5", + }); + const result = await runExecuteTraced(limiter, recorded); + + expect(result).toMatchObject({ result: "ran" }); + + const check = spanNamed(recorded, "rate_limit.check"); + expect(check.attributes.get("rate_limit.check.timed_out")).toBe(true); + expect(check.attributes.get("rate_limit.check.error_tag")).toBe("RateLimitCheckTimeoutError"); + }); + + // The healthy check's span (count / limit / blocked / failed_open) is NOT + // asserted here: the cloud e2e scenario "the rate-limit counter check is + // visible in the exported spans" pins it on the real workerd + Durable + // Object topology, against the spans the worker actually exports. The two + // cases above stay because neither is reachable from the e2e harness — it + // has no fault seam for a Durable Object RPC, and the check budget is a + // process-wide worker var that the shared cloud boot cannot vary per + // scenario without disabling the backstop for the whole run. +}); + +// --------------------------------------------------------------------------- +// Counter Durable Object +// --------------------------------------------------------------------------- + +/** Minimal DO storage that counts the calls the increment path makes. */ +const fakeStorage = () => { + const values = new Map(); + const calls = { put: 0, setAlarm: 0, deleteAll: 0 }; + return { + calls, + storage: { + get: (key: string) => Promise.resolve(values.get(key)), + put: (key: string, value: unknown) => { + calls.put += 1; + values.set(key, value); + return Promise.resolve(); + }, + setAlarm: () => { + calls.setAlarm += 1; + return Promise.resolve(); + }, + deleteAll: () => { + calls.deleteAll += 1; + values.clear(); + return Promise.resolve(); + }, + }, + }; +}; + +const makeCounter = (storage: ReturnType["storage"]) => + // oxlint-disable-next-line executor/no-double-cast -- test double: only the four storage methods the counter uses are implemented + new ExecutionRateLimiterDO({ storage } as unknown as DurableObjectState, {} as Env); + +describe("execution rate-limit counter DO", () => { + it("writes the purge alarm once per window instead of on every increment", async () => { + // The alarm only has to outlive the window; rewriting it on every call put + // a second durable write on the hot path with the input gate closed. + const fake = fakeStorage(); + const counter = makeCounter(fake.storage); + + expect(await counter.increment(7)).toBe(1); + expect(await counter.increment(7)).toBe(2); + expect(await counter.increment(7)).toBe(3); + + expect(fake.calls.put, "every increment still persists the count").toBe(3); + expect(fake.calls.setAlarm, "the purge alarm is written once for the window").toBe(1); + }); + + it("moves the purge alarm when the window rolls", async () => { + const fake = fakeStorage(); + const counter = makeCounter(fake.storage); + + await counter.increment(7); + await counter.increment(7); + expect(await counter.increment(8), "a new window restarts the count").toBe(1); + + expect(fake.calls.setAlarm, "one alarm write per window, not per increment").toBe(2); + }); + + it("re-arms the purge alarm after it has fired", async () => { + const fake = fakeStorage(); + const counter = makeCounter(fake.storage); + + await counter.increment(7); + await counter.alarm(); + expect(fake.calls.deleteAll, "the alarm purges the counter's storage").toBe(1); + + expect(await counter.increment(7), "the purge reset the window's count").toBe(1); + expect(fake.calls.setAlarm, "a purged counter schedules a fresh purge").toBe(2); + }); +}); diff --git a/apps/cloud/src/engine/execution-rate-limit.ts b/apps/cloud/src/engine/execution-rate-limit.ts index b68d4c044..906bee252 100644 --- a/apps/cloud/src/engine/execution-rate-limit.ts +++ b/apps/cloud/src/engine/execution-rate-limit.ts @@ -1,5 +1,5 @@ // --------------------------------------------------------------------------- -// Per-org execution rate limit — an abuse backstop independent of billing. +// Per-org execution rate limit — a free-tier abuse backstop. // // The balance gate (execution-gate.ts) depends on Autumn and fails open, so a // billing outage plus runaway automation could still run unbounded executions. @@ -8,13 +8,26 @@ // each MCP session lives in its own DO instance, so an in-memory counter // would be per-session and trivially bypassed by opening more sessions). // -// Like the balance gate it FAILS OPEN: an unreachable counter DO, a missing +// Paid organizations are EXEMPT. The cap was sized for free-tier abuse but +// applied to everyone, and on 2026-08-18 it blocked a paying customer +// mid-workload — their agent gave up on Executor and routed around it. Paid +// usage is what the balance gate and metered overage are for; this backstop +// has no business capping it. +// +// The exemption is resolved ONLY once the counter reports an org over the cap, +// so the common path (under the cap) costs the counter increment and nothing +// else. `isExempt` is an opaque predicate: this module still names no billing +// concept, and the Autumn coupling lives in `execution-stack-metered.ts`, +// which already owns that dependency. +// +// FAIL OPEN applies to the COUNTER: an unreachable counter DO, a missing // binding, or a slow call allows the execution (warn + Sentry). The backstop -// must never take executions down with it. +// must never take executions down with it. An unresolved EXEMPTION is the one +// thing that does not fail open — see `resolveExemption`. // --------------------------------------------------------------------------- import { DurableObject, env } from "cloudflare:workers"; -import { Data, Effect } from "effect"; +import { Data, Effect, Predicate } from "effect"; import type * as Cause from "effect/Cause"; import type { ExecutionEngine } from "@executor-js/execution"; @@ -25,14 +38,26 @@ import { RATE_LIMIT_BLOCKED_MESSAGE } from "./execution-limit-messages"; // Fixed window: all executions in the same clock hour share one counter. export const RATE_LIMIT_WINDOW_MS = 3_600_000; -// Calibration: the heaviest legitimate org runs ~1.1k executions per MONTH, -// so 1000 per HOUR is far above any human-driven usage and only trips on -// runaway automation (the incident this backstops: ~18k in 30 days would -// still pass, which is fine — that class of overrun is the balance gate's -// job; this catches tight loops). +// The cap for organizations WITHOUT a paid subscription. +// +// The original calibration ("the heaviest legitimate org runs ~1.1k executions +// per MONTH, so 1000 per HOUR is far above any human-driven usage") went stale +// inside six weeks: by 2026-08-18 a paying org was sustaining ~5.3k executions +// per DAY and crossed this cap in a single hour. Sizing a shared number +// against the largest customer is a losing game, so the number no longer tries +// to describe them — paid orgs are exempt below, and this now has only +// free-tier abuse to cover, which is what it was picked for. export const EXECUTIONS_PER_ORG_PER_HOUR = 1000; // Counter DO slower than this => fail open rather than stall executions. const RATE_LIMIT_CHECK_TIMEOUT_MS = 2_000; +// Exemption lookup slower than this => treat as unresolved. +const EXEMPTION_CHECK_TIMEOUT_MS = 2_000; +// An org over the cap is checked at most once per TTL rather than once per +// execution, so a paid org running far past it doesn't hammer the lookup. +const EXEMPTION_CACHE_TTL_MS = 60_000; +// Sweep guard, mirroring the balance gate's cache: one long-lived isolate can +// serve many orgs. +const EXEMPTION_CACHE_MAX_ENTRIES = 10_000; // The DO purges its storage this long after the last increment, so idle orgs // cost nothing. Two windows: long enough that an active window never purges. const COUNTER_PURGE_AFTER_MS = 2 * RATE_LIMIT_WINDOW_MS; @@ -51,6 +76,80 @@ class RateLimitCheckTimeoutError extends Data.TaggedError("RateLimitCheckTimeout readonly timeoutMs: number; }> {} +/** + * Why a counter DO call failed, as a small closed vocabulary. + * + * The counter's failures are overwhelmingly transient Cloudflare platform + * faults, and they used to arrive at error reporting as one untyped + * `UnknownError: An error occurred in Effect.tryPromise` with no application + * frames — a group that says nothing and would eventually swallow a real + * misconfiguration too. The code is what makes a storage reset (retryable, + * expected) distinguishable from an overload or an outright unknown fault. + */ +export type RateLimitCounterErrorCode = + | "storage_reset" + | "overloaded" + | "exceeded_memory" + | "network" + | "unknown"; + +/** A counter DO call that failed, carrying the classification and the org. */ +export class RateLimitCounterError extends Data.TaggedError("RateLimitCounterError")<{ + readonly organizationId: string; + readonly code: RateLimitCounterErrorCode; + readonly reason: string; + readonly cause: unknown; +}> {} + +// Cloudflare surfaces these as plain `Error`s with documented message text; +// there is no structured code to read, so the message is the only signal. +// (A shared `classifyDurableObjectError` would be the right home for this once +// one exists.) +const counterErrorCode = (reason: string): RateLimitCounterErrorCode => { + if (/caused object to be reset/i.test(reason)) return "storage_reset"; + if (/overloaded/i.test(reason)) return "overloaded"; + if (/exceeded (its )?memory|out of memory/i.test(reason)) return "exceeded_memory"; + if (/network connection lost|connection.*(lost|reset)/i.test(reason)) return "network"; + return "unknown"; +}; + +/** + * The fail-open landing: record the outcome on the check span, warn, and allow + * the execution. Only failures that are NOT deliberate degradation reach the + * error reporter — see the call sites in `decide`. + */ +const failOpen = ( + error: unknown, + outcome: { readonly errorTag: string; readonly timedOut: boolean }, +): Effect.Effect => + Effect.gen(function* () { + yield* Effect.annotateCurrentSpan({ + "rate_limit.blocked": false, + "rate_limit.check.failed_open": true, + "rate_limit.check.timed_out": outcome.timedOut, + "rate_limit.check.error_tag": outcome.errorTag, + }); + yield* Effect.sync(() => { + console.warn("[rate-limit] execution rate limit check failed open:", error); + }); + if (!outcome.timedOut) yield* captureCauseEffect(error); + return { blocked: false } as const satisfies GateDecision; + }); + +/** Internal sentinel for an exemption lookup that exceeded its time budget. */ +class ExemptionCheckTimeoutError extends Data.TaggedError("ExemptionCheckTimeoutError")<{ + readonly timeoutMs: number; +}> {} + +/** + * Whether an organization is exempt from the cap. Production passes a paid- + * subscription check; keeping it an opaque predicate is what lets this module + * stay free of any billing import. + */ +export type ExecutionRateLimitExemption = ( + organizationId: string, +) => Effect.Effect; + // --------------------------------------------------------------------------- // Counter Durable Object — one instance per organization (idFromName(orgId)). // Stores a single { windowId, count } record: an increment in a new window @@ -67,6 +166,12 @@ type WindowRecord = { export class ExecutionRateLimiterDO extends DurableObject { private readonly counterStorage: DurableObjectState["storage"]; + /** + * The window this instance has already armed the purge alarm for. In-memory + * on purpose: it costs no storage read, and a fresh instance (eviction, cold + * start) simply re-arms on its first increment. + */ + private purgeArmedForWindow: number | null = null; constructor(ctx: DurableObjectState, doEnv: Env) { super(ctx, doEnv); @@ -80,11 +185,21 @@ export class ExecutionRateLimiterDO extends DurableObject { const stored = await this.counterStorage.get(WINDOW_RECORD_KEY); const count = stored && stored.windowId === windowId ? stored.count + 1 : 1; await this.counterStorage.put(WINDOW_RECORD_KEY, { windowId, count }); - await this.counterStorage.setAlarm(Date.now() + COUNTER_PURGE_AFTER_MS); + // The alarm only has to outlive the window, and it is set two windows out, + // so once per window is enough — rewriting it on every increment put a + // second durable write and an alarm-manager update on the hot path of + // every execution, with the input gate closed across all three. `count` + // back at 1 means the window rolled (or a purge already ran), so the + // deadline moves with it. + if (count === 1 || this.purgeArmedForWindow !== windowId) { + await this.counterStorage.setAlarm(Date.now() + COUNTER_PURGE_AFTER_MS); + this.purgeArmedForWindow = windowId; + } return count; } async alarm(): Promise { + this.purgeArmedForWindow = null; await this.counterStorage.deleteAll(); } } @@ -93,11 +208,17 @@ export class ExecutionRateLimiterDO extends DurableObject { // Client // --------------------------------------------------------------------------- -/** Count one execution for (organizationId, windowId); returns the new count. */ +/** + * Count one execution for (organizationId, windowId); returns the new count. + * + * The failure channel is typed rather than `unknown` so the fail-open path can + * tell a counter fault from a blown budget by tag, and so error reporting + * groups by cause instead of by one opaque `UnknownError`. + */ export type RateLimitIncrement = ( organizationId: string, windowId: number, -) => Effect.Effect; +) => Effect.Effect; export type ExecutionRateLimiter = { readonly decorate: ( @@ -119,12 +240,76 @@ export const makeExecutionRateLimiter = ( readonly windowMs?: number; readonly timeoutMs?: number; readonly now?: () => number; + readonly isExempt?: ExecutionRateLimitExemption; + readonly exemptionTtlMs?: number; }, ): ExecutionRateLimiter => { const limit = options?.limit ?? EXECUTIONS_PER_ORG_PER_HOUR; const windowMs = options?.windowMs ?? RATE_LIMIT_WINDOW_MS; const timeoutMs = options?.timeoutMs ?? RATE_LIMIT_CHECK_TIMEOUT_MS; const now = options?.now ?? Date.now; + const isExempt = options?.isExempt; + const exemptionTtlMs = options?.exemptionTtlMs ?? EXEMPTION_CACHE_TTL_MS; + + const exemptionCache = new Map< + string, + { readonly exempt: boolean; readonly expiresAtMs: number } + >([]); + + const writeExemptionCache = (organizationId: string, exempt: boolean, nowMs: number): void => { + if (exemptionCache.size >= EXEMPTION_CACHE_MAX_ENTRIES) { + for (const [key, entry] of exemptionCache) { + if (entry.expiresAtMs <= nowMs) exemptionCache.delete(key); + } + // Still saturated after dropping expired entries: reset rather than grow. + if (exemptionCache.size >= EXEMPTION_CACHE_MAX_ENTRIES) exemptionCache.clear(); + } + exemptionCache.set(organizationId, { exempt, expiresAtMs: nowMs + exemptionTtlMs }); + }; + + /** + * Resolved only for orgs already over the cap, so the lookup never touches + * the common path. + * + * This is the one place that does NOT fail open. The balance gate already + * allows executions when Autumn is unreachable; if the exemption did too, + * an Autumn outage would switch this backstop off entirely — precisely the + * "billing outage plus runaway automation" case it exists to cover. A stale + * positive is honoured ahead of that fallback, so a blip can't flip a + * known-paid org into a block mid-workload. + */ + const resolveExemption = (organizationId: string): Effect.Effect => + Effect.suspend(() => { + if (!isExempt) return Effect.succeed(false); + const nowMs = now(); + const cached = exemptionCache.get(organizationId); + if (cached && cached.expiresAtMs > nowMs) return Effect.succeed(cached.exempt); + return isExempt(organizationId).pipe( + Effect.timeoutOrElse({ + duration: `${EXEMPTION_CHECK_TIMEOUT_MS} millis`, + orElse: () => + Effect.fail(new ExemptionCheckTimeoutError({ timeoutMs: EXEMPTION_CHECK_TIMEOUT_MS })), + }), + Effect.map((exempt) => { + writeExemptionCache(organizationId, exempt, nowMs); + return exempt; + }), + Effect.catch((error: unknown) => + Effect.gen(function* () { + yield* Effect.sync(() => { + console.warn( + `[rate-limit] exemption lookup failed for ${organizationId}; treating as ${ + cached ? "last known" : "not exempt" + }:`, + error, + ); + }); + yield* captureCauseEffect(error); + return cached?.exempt ?? false; + }), + ), + ); + }); const decide = (organizationId: string): Effect.Effect => Effect.suspend(() => { @@ -134,29 +319,86 @@ export const makeExecutionRateLimiter = ( duration: `${timeoutMs} millis`, orElse: () => Effect.fail(new RateLimitCheckTimeoutError({ timeoutMs })), }), - Effect.map( - (count): GateDecision => - count > limit - ? { - blocked: true, - error: new ExecutionRateLimitExceededError({ - organizationId, - message: RATE_LIMIT_BLOCKED_MESSAGE, - }), - } - : { blocked: false }, - ), + Effect.flatMap((count): Effect.Effect => { + // Under the cap: no exemption lookup, no extra I/O. + if (count <= limit) + return Effect.as( + Effect.annotateCurrentSpan({ + "rate_limit.count": count, + "rate_limit.blocked": false, + "rate_limit.check.failed_open": false, + }), + { blocked: false }, + ); + return Effect.gen(function* () { + yield* Effect.annotateCurrentSpan({ + "rate_limit.count": count, + "rate_limit.check.failed_open": false, + }); + if (yield* resolveExemption(organizationId)) { + yield* Effect.annotateCurrentSpan({ + "rate_limit.blocked": false, + "rate_limit.exempt": true, + }); + return { blocked: false } as const satisfies GateDecision; + } + // The only record that the backstop fired. A blocked execution is + // never usage-tracked (the gate short-circuits before the tracker) + // and deliberately not sent to Sentry — a backstop stopping + // runaway automation is expected, not exceptional — so without + // this line a blocked org is invisible outside a bug report, which + // is how the 2026-08-18 block went unnoticed until a customer + // sent a screenshot. + yield* Effect.sync(() => { + console.warn( + `[rate-limit] blocked execution for ${organizationId}: ${count} > ${limit} in window ${windowId}`, + ); + }); + yield* Effect.annotateCurrentSpan({ + "rate_limit.blocked": true, + "rate_limit.exempt": false, + }); + return { + blocked: true, + error: new ExecutionRateLimitExceededError({ + organizationId, + message: RATE_LIMIT_BLOCKED_MESSAGE, + }), + } as const satisfies GateDecision; + }); + }), // FAIL OPEN: the backstop must never block executions because its // counter is unreachable or slow. + // + // A check that blew its own budget is DELIBERATE degradation, not an + // exception — the timeout exists precisely so a slow counter can't + // stall a user-facing execution. It is measured on the span + // (`rate_limit.check.timed_out`), where a step change in the rate is + // alertable, rather than paged per occurrence, which is what buried + // real counter failures under one opaque group. Everything else (RPC + // faults, a missing binding) still reports. + Effect.catchTag("RateLimitCheckTimeoutError", (error) => + failOpen(error, { errorTag: "RateLimitCheckTimeoutError", timedOut: true }), + ), + // A catch-all rather than a second `catchTag`: fail-open is a hard + // requirement and must not depend on the failure being one the types + // predicted. Effect.catch((error: unknown) => - Effect.gen(function* () { - yield* Effect.sync(() => { - console.warn("[rate-limit] execution rate limit check failed open:", error); - }); - yield* captureCauseEffect(error); - return { blocked: false } as const satisfies GateDecision; + failOpen(error, { + errorTag: Predicate.isTagged(error, "RateLimitCounterError") + ? "RateLimitCounterError" + : "unknown", + timedOut: false, }), ), + Effect.withSpan("rate_limit.check", { + attributes: { + "rate_limit.organization_id": organizationId, + "rate_limit.window_id": windowId, + "rate_limit.limit": limit, + "rate_limit.check.timeout_ms": timeoutMs, + }, + }), ); }); @@ -185,8 +427,13 @@ type RateLimiterNamespace = { * Production rate limiter backed by the `EXECUTION_RATE_LIMITER` counter DO. * When the binding is absent (unit-test workers, older local setups) the * limiter is disabled: every check passes, logged once at construction. + * + * `isExempt` decides which orgs the cap skips; production passes a paid- + * subscription check from `execution-stack-metered.ts`. */ -export const makeCloudExecutionRateLimiter = (): ExecutionRateLimiter => { +export const makeCloudExecutionRateLimiter = ( + isExempt: ExecutionRateLimitExemption, +): ExecutionRateLimiter => { const limit = resolveRateLimit(); const namespace = (env as { EXECUTION_RATE_LIMITER?: RateLimiterNamespace }) .EXECUTION_RATE_LIMITER; @@ -196,17 +443,53 @@ export const makeCloudExecutionRateLimiter = (): ExecutionRateLimiter => { ); return makeExecutionRateLimiter(() => Effect.succeed(0)); } - return makeExecutionRateLimiter( - (organizationId, windowId) => - Effect.tryPromise(() => { + return makeExecutionRateLimiter(counterIncrement(namespace), { + limit, + timeoutMs: resolveCheckTimeoutMs(), + isExempt, + }); +}; + +/** + * The counter DO RPC, as a traced and typed increment. + * + * The span is the whole point: this call is a blocking, cold-startable hop on + * the execute hot path, and until it had one nothing about its duration was + * measurable — the only evidence it was slow was the fail-open warning 2s + * later. The typed error replaces Effect's generic `UnknownError`, which + * reported no org, no window, and no hint that a Durable Object was involved. + */ +const counterIncrement = + (namespace: RateLimiterNamespace): RateLimitIncrement => + (organizationId, windowId) => + Effect.tryPromise({ + try: () => { const stub = namespace.get( namespace.idFromName(organizationId), ) as ExecutionRateLimiterStub; return stub.increment(windowId); + }, + catch: (cause) => { + // oxlint-disable-next-line executor/no-instanceof-error, executor/no-unknown-error-message -- boundary: the Durable Object RPC rejects with a plain platform Error whose message text is the only classification signal Cloudflare gives + const reason = cause instanceof Error ? cause.message : String(cause); + return new RateLimitCounterError({ + organizationId, + code: counterErrorCode(reason), + reason, + cause, + }); + }, + }).pipe( + Effect.tapError((error) => + Effect.annotateCurrentSpan({ + "rate_limit.counter.error_tag": "RateLimitCounterError", + "rate_limit.counter.error_code": error.code, + }), + ), + Effect.withSpan("rate_limit.increment", { + attributes: { "rate_limit.window_id": windowId }, }), - { limit }, - ); -}; + ); /** * The per-org hourly cap: the `EXECUTION_RATE_LIMIT_PER_HOUR` env override @@ -220,3 +503,18 @@ const resolveRateLimit = (): number => { const parsed = Number.parseInt(raw, 10); return Number.isInteger(parsed) && parsed > 0 ? parsed : EXECUTIONS_PER_ORG_PER_HOUR; }; + +/** + * The counter's time budget: the `EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS` env + * override or `RATE_LIMIT_CHECK_TIMEOUT_MS` when it's unset or unparseable. + * Same precedent and same purpose as `EXECUTION_RATE_LIMIT_PER_HOUR`: the + * production 2s budget can't be blown on demand, so tests set a tiny one to + * drive the fail-open path deterministically. Production leaves it unset. + */ +const resolveCheckTimeoutMs = (): number => { + const raw = (env as { EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS?: string }) + .EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS; + if (raw === undefined) return RATE_LIMIT_CHECK_TIMEOUT_MS; + const parsed = Number.parseInt(raw, 10); + return Number.isInteger(parsed) && parsed > 0 ? parsed : RATE_LIMIT_CHECK_TIMEOUT_MS; +}; diff --git a/apps/cloud/src/engine/execution-stack-metered.ts b/apps/cloud/src/engine/execution-stack-metered.ts index 4ea70b6d4..6063e1456 100644 --- a/apps/cloud/src/engine/execution-stack-metered.ts +++ b/apps/cloud/src/engine/execution-stack-metered.ts @@ -26,6 +26,7 @@ import { } from "@executor-js/api/server"; import { AutumnService } from "../extensions/billing/service"; +import { hasPaidOrganizationSubscription } from "../extensions/billing/plans"; import type { DbService } from "../db/db"; import { CloudExecutionSeamsLayer } from "../engine/execution-stack"; import { makeExecutionLimitGate } from "./execution-gate"; @@ -34,7 +35,8 @@ import { withExecutionUsageTracking } from "./execution-usage"; // Usage-metering decorator bound to the billing service, plus the two // pre-execution guards this layer owns, ordered cheapest first: -// 1. rate-limit backstop (counter DO, independent of billing) +// 1. rate-limit backstop (counter DO; free-tier abuse only — paid orgs are +// exempt via the subscription lookup below) // 2. execution balance gate (Autumn check, cached 60s, fails open) // 3. usage tracking — fire-and-forget (`Effect.runFork`) so the billing // call can't stall a user-facing execution. @@ -48,7 +50,18 @@ export const CloudMeteringEngineDecorator: Layer.Layer autumn.checkExecutionBalance(organizationId), ); - const rateLimiter = makeCloudExecutionRateLimiter(); + // The limiter's paid-org exemption. This is the billing coupling the + // limiter module deliberately avoids owning, and it reads the same + // `PAID_AUTUMN_PLAN_IDS` config as the org-creation and seat gates so + // "paid" means one thing across the app. The limiter calls this only + // for orgs already over the cap and caches the answer, so the extra + // Autumn round trip stays off the hot path. + const rateLimiter = makeCloudExecutionRateLimiter((organizationId) => + Effect.map( + autumn.use((client) => client.customers.getOrCreate({ customerId: organizationId })), + (customer) => hasPaidOrganizationSubscription(customer.subscriptions), + ), + ); return { decorate: (engine, identity: EngineStackIdentity) => rateLimiter.decorate( diff --git a/apps/cloud/src/engine/execution-stack.ts b/apps/cloud/src/engine/execution-stack.ts index 869bf5816..a7ff4de70 100644 --- a/apps/cloud/src/engine/execution-stack.ts +++ b/apps/cloud/src/engine/execution-stack.ts @@ -42,10 +42,17 @@ import { PluginsProvider, collectTables, } from "@executor-js/api/server"; +import { googleCatalogOAuthScopesForPreset } from "@executor-js/plugin-openapi/providers/google"; +import { slackMcpUserScopes } from "@executor-js/react/lib/slack-mcp-oauth"; import { makeDynamicWorkerExecutor } from "@executor-js/runtime-dynamic-worker"; -import type { AnyPlugin } from "@executor-js/sdk"; +import { + IntegrationSlug, + type AnyPlugin, + type FirstPartyOAuthClientConfig, +} from "@executor-js/sdk"; import executorConfig from "../../executor.config"; +import { cloudEnterpriseManagedRollout } from "../analytics/ema-rollout"; import { DbService } from "../db/db"; import { cloudDbProviderLayer } from "../db/fuma"; @@ -88,6 +95,98 @@ export const CloudPluginsProvider: Layer.Layer = Layer.succeed( */ export const CLOUD_MOUNT_PREFIX = "/api" as const; +// Consumer Google launch boundary. Keep this list aligned with the scopes +// submitted for the Executor-owned production app: ordinary Workspace services +// plus Photos, Meet, and Search Console. Admin, Classroom, YouTube, Apps Script, +// BigQuery, and Cloud Resource Manager have materially different audiences or +// provider requirements and remain BYO OAuth. The same scope source builds each +// catalog auth template, preventing picker/start drift. +const GOOGLE_FIRST_PARTY_PRESET_IDS = [ + "google-calendar", + "google-meet", + "google-gmail", + "google-sheets", + "google-drive", + "google-docs", + "google-slides", + "google-forms", + "google-tasks", + "google-people", + "google-photos-library", + "google-photos-picker", + "google-search-console", +] as const; + +const GOOGLE_FIRST_PARTY_ALLOWED_SCOPES: readonly string[] = [ + ...new Set([ + ...GOOGLE_FIRST_PARTY_PRESET_IDS.flatMap(googleCatalogOAuthScopesForPreset), + // Connections created before the full-Gmail review retain this declared + // scope on reconnect. New Gmail presets request `mail.google.com`. + "https://www.googleapis.com/auth/gmail.modify", + ]), +]; + +// Executor-owned provider apps, enabled per provider by setting BOTH env vars +// (id + secret). Each provider-side registration must list +// `${VITE_PUBLIC_SITE_URL}/api/oauth/callback` as its callback; the org slug +// travels inside OAuth `state`, so the single static callback serves every org. +// +// The endpoint URLs default to the real provider; the `_AUTHORIZE_URL` / +// `_TOKEN_URL` overrides exist so tests and dev instances can point the app at +// an emulated provider (`@executor-js/emulate`) and run the complete flow. +// Production leaves them unset. +export const cloudFirstPartyOAuthClients = (): readonly FirstPartyOAuthClientConfig[] => [ + ...(env.FIRST_PARTY_GITHUB_CLIENT_ID && env.FIRST_PARTY_GITHUB_CLIENT_SECRET + ? [ + { + name: "github", + authorizationUrl: + env.FIRST_PARTY_GITHUB_AUTHORIZE_URL ?? "https://github.com/login/oauth/authorize", + tokenUrl: + env.FIRST_PARTY_GITHUB_TOKEN_URL ?? "https://github.com/login/oauth/access_token", + clientId: env.FIRST_PARTY_GITHUB_CLIENT_ID, + clientSecret: env.FIRST_PARTY_GITHUB_CLIENT_SECRET, + integrations: [IntegrationSlug.make("github_rest")], + // GitHub App user access tokens do not use classic OAuth scopes; + // their capabilities come from the app's registered permissions. + authorizationScopes: [], + }, + ] + : []), + ...(env.FIRST_PARTY_GOOGLE_CLIENT_ID && env.FIRST_PARTY_GOOGLE_CLIENT_SECRET + ? [ + { + name: "google", + authorizationUrl: "https://accounts.google.com/o/oauth2/v2/auth", + tokenUrl: "https://oauth2.googleapis.com/token", + clientId: env.FIRST_PARTY_GOOGLE_CLIENT_ID, + clientSecret: env.FIRST_PARTY_GOOGLE_CLIENT_SECRET, + allowedScopes: GOOGLE_FIRST_PARTY_ALLOWED_SCOPES, + // Withdrawn from the connect picker: no new connection is offered the + // Executor-owned Google app. The entry stays declared on purpose — + // every connection already minted against it keeps refreshing and + // reconnecting through it. Deleting this block, or unsetting the env + // vars, would strand those connections instead. + unlisted: true, + }, + ] + : []), + ...(env.FIRST_PARTY_SLACK_CLIENT_ID && env.FIRST_PARTY_SLACK_CLIENT_SECRET + ? [ + { + name: "slack", + authorizationUrl: "https://slack.com/oauth/v2_user/authorize", + tokenUrl: "https://slack.com/api/oauth.v2.user.access", + resource: "https://mcp.slack.com", + clientId: env.FIRST_PARTY_SLACK_CLIENT_ID, + clientSecret: env.FIRST_PARTY_SLACK_CLIENT_SECRET, + integrations: [IntegrationSlug.make("slack")], + allowedScopes: slackMcpUserScopes, + }, + ] + : []), +]; + export const CloudHostConfig: Layer.Layer = Layer.sync(HostConfig, () => ({ // SSRF / private-network egress guard. Config-driven, NOT a test flag: // production leaves `ALLOW_LOCAL_NETWORK` unset so the guard stays ON (`false`); @@ -99,6 +198,12 @@ export const CloudHostConfig: Layer.Layer = Layer.sync(HostConfig, ( // WorkOS Vault is cloud's credential storage implementation detail, not a // user-selectable provider surface. exposeCredentialProviders: false, + firstPartyOAuthClients: cloudFirstPartyOAuthClients(), + // Enterprise-managed authorization ships behind a PostHog flag. Cloud is the + // one host with a flag service, so cloud is the one host that installs a + // gate; everywhere else the seam stays empty and the profile is attempted as + // before. Gating happens at connect only — see the SDK contract. + enterpriseManagedRollout: cloudEnterpriseManagedRollout(), })); export const CloudCodeExecutorProvider: Layer.Layer = Layer.sync( diff --git a/apps/cloud/src/engine/first-party-oauth-clients.test.ts b/apps/cloud/src/engine/first-party-oauth-clients.test.ts new file mode 100644 index 000000000..9780a1c62 --- /dev/null +++ b/apps/cloud/src/engine/first-party-oauth-clients.test.ts @@ -0,0 +1,87 @@ +import { env } from "cloudflare:workers"; +import { beforeAll, describe, expect, it } from "@effect/vitest"; + +import { cloudFirstPartyOAuthClients } from "./execution-stack"; + +// The reviewed consumer scope boundary of the Executor-owned Google app. +// +// These assertions used to live in `e2e/scenarios/first-party-oauth.test.ts`, +// read off `listClients`. The app is now `unlisted`, so it has no read surface +// to introspect — the bundle is only observable on the config it is built from, +// which is here. The e2e still owns the BEHAVIOUR the boundary produces (which +// scopes an `oauth.start` requests, and that admin scopes are refused). +const GOOGLE_SCOPE = (suffix: string) => `https://www.googleapis.com/auth/${suffix}`; + +describe("cloud first-party oauth clients", () => { + beforeAll(() => { + env.FIRST_PARTY_GOOGLE_CLIENT_ID = "test-google-client"; + env.FIRST_PARTY_GOOGLE_CLIENT_SECRET = "test-google-secret"; + }); + + const google = () => cloudFirstPartyOAuthClients().find((client) => client.name === "google"); + + it("declares the Google app but withholds it from every listing", () => { + const client = google(); + expect(client, "the env-declared first-party Google app is configured").toBeDefined(); + // The entry MUST stay declared: `loadClient` resolves it by slug for every + // existing connection's refresh and reconnect. `unlisted` is what stops it + // being offered for new connections. + expect(client?.unlisted).toBe(true); + }); + + it("covers the reviewed consumer bundle", () => { + const allowed = google()?.allowedScopes; + expect(allowed).toBeDefined(); + for (const scope of [ + "calendar", + "meetings.space.readonly", + "spreadsheets", + "drive.file", + "drive", + "documents", + "presentations", + "forms.body", + "forms.responses.readonly", + "tasks", + "contacts", + "contacts.other.readonly", + "directory.readonly", + "user.addresses.read", + "user.birthday.read", + "user.emails.read", + "user.gender.read", + "user.organization.read", + "user.phonenumbers.read", + "photoslibrary.appendonly", + "photoslibrary.edit.appcreateddata", + "photospicker.mediaitems.readonly", + "webmasters", + "gmail.settings.basic", + ]) { + expect(allowed).toContain(GOOGLE_SCOPE(scope)); + } + // `gmail.modify` stays in the host-enforced allowlist on purpose: a + // connection created before the full-Gmail review still declares it, and + // `resolveFirstPartyScopes` filters discovered scopes through this list, so + // dropping it would break those reconnects — as the legacy-spec case in the + // e2e asserts. The invariant that new Gmail presets request + // `mail.google.com` instead lives in the preset unit tests + // (packages/plugins/openapi/.../presets.test.ts), which is where the + // request-side scope choice is actually decided. + expect(allowed).toContain("https://mail.google.com/"); + expect(allowed).toContain(GOOGLE_SCOPE("gmail.modify")); + }); + + it("excludes the scopes held back from consumer review", () => { + const allowed = google()?.allowedScopes; + expect(allowed).toBeDefined(); + for (const scope of [ + "gmail.settings.sharing", + "admin.directory.user", + "youtube", + "cloud-platform", + ]) { + expect(allowed).not.toContain(GOOGLE_SCOPE(scope)); + } + }); +}); diff --git a/apps/cloud/src/env-augment.d.ts b/apps/cloud/src/env-augment.d.ts index 4ef130bfa..bb31aa00a 100644 --- a/apps/cloud/src/env-augment.d.ts +++ b/apps/cloud/src/env-augment.d.ts @@ -6,6 +6,13 @@ declare global { namespace Cloudflare { interface Env { // Observability + // Worker version metadata binding (wrangler.jsonc `version_metadata`). + // Optional so test workers and local setups without the binding still + // typecheck; spans then carry the "dev" service.version default. + CF_VERSION_METADATA?: WorkerVersionMetadata; + // Commit that produced the running deploy, passed by CI as + // `wrangler deploy --var GIT_COMMIT_SHA:$GITHUB_SHA`. Absent outside CI. + GIT_COMMIT_SHA?: string; AXIOM_TOKEN?: string; AXIOM_DATASET?: string; AXIOM_TRACES_URL?: string; @@ -53,6 +60,30 @@ declare global { // number to drive the backstop. Production leaves it unset. EXECUTION_RATE_LIMIT_PER_HOUR?: string; + // Optional override for the counter DO's check budget in milliseconds + // (defaults to RATE_LIMIT_CHECK_TIMEOUT_MS = 2000 when unset or + // unparseable). Same purpose as the cap override: the production budget + // can't be blown on demand, so tests set a tiny one to exercise the + // fail-open path. Production leaves it unset. + EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS?: string; + + // First-party OAuth apps (executor-owned provider registrations). Each + // pair enables one-click connect through `first-party:`; an + // unset pair simply ships no first-party app for that provider. The + // registered callback on the provider side must be + // `${VITE_PUBLIC_SITE_URL}/api/oauth/callback`. + FIRST_PARTY_GITHUB_CLIENT_ID?: string; + FIRST_PARTY_GITHUB_CLIENT_SECRET?: string; + // Endpoint overrides for the GitHub first-party app, so tests/dev can + // point it at an emulated provider and complete the whole flow. Unset in + // production (the real github.com endpoints are the defaults). + FIRST_PARTY_GITHUB_AUTHORIZE_URL?: string; + FIRST_PARTY_GITHUB_TOKEN_URL?: string; + FIRST_PARTY_GOOGLE_CLIENT_ID?: string; + FIRST_PARTY_GOOGLE_CLIENT_SECRET?: string; + FIRST_PARTY_SLACK_CLIENT_ID?: string; + FIRST_PARTY_SLACK_CLIENT_SECRET?: string; + // Billing AUTUMN_SECRET_KEY?: string; /** Optional Autumn base-URL override (Autumn emulator in tests/dev). */ diff --git a/apps/cloud/src/extensions/billing/route.node.test.ts b/apps/cloud/src/extensions/billing/route.node.test.ts index f6274facc..dc2a5a731 100644 --- a/apps/cloud/src/extensions/billing/route.node.test.ts +++ b/apps/cloud/src/extensions/billing/route.node.test.ts @@ -34,7 +34,7 @@ const stubWorkOS = Layer.succeed( ); const stubUsers = Layer.succeed(UserStoreService)({ - use: (fn) => + use: (_op, fn) => Effect.promise(() => fn({ ensureAccount: async (id: string) => ({ id, createdAt }), diff --git a/apps/cloud/src/extensions/billing/service.ts b/apps/cloud/src/extensions/billing/service.ts index 1b33b203c..e96c8ed84 100644 --- a/apps/cloud/src/extensions/billing/service.ts +++ b/apps/cloud/src/extensions/billing/service.ts @@ -17,15 +17,61 @@ export class AutumnError extends Data.TaggedError("AutumnError")<{ cause?: unknown; }> {} +/** + * Autumn has no customer record for the organization. Split out from + * `AutumnError` because it is not an outage and must not be treated like one: + * an outage is transient and the right answer is to fail open and page, while + * a missing customer is a PERMANENT provisioning gap — every subsequent + * balance call 404s the same way, so the org runs unbilled and unmetered + * forever. Callers that own an organization id repair it (see + * `withProvisionedCustomer`); everything else still surfaces as `AutumnError`. + */ +export class AutumnCustomerNotFoundError extends Data.TaggedError("AutumnCustomerNotFoundError")<{ + message: string; + cause?: unknown; +}> {} + +export type AutumnFailure = AutumnError | AutumnCustomerNotFoundError; + +// Autumn's own error code for "no such customer", carried in the JSON body of +// a 404 (`{"message":"Customer not found","code":"customer_not_found"}`). +const CUSTOMER_NOT_FOUND_CODE = "customer_not_found"; + +/** + * True when `cause` is Autumn's "no such customer" answer. The autumn-js SDK + * throws an `AutumnError` carrying the raw HTTP `statusCode` and `body`; match + * on the code rather than the status alone so an unrelated 404 (a removed + * endpoint, a proxy) is still reported as a genuine failure. + */ +const isCustomerNotFoundCause = (cause: unknown): boolean => { + if (typeof cause !== "object" || cause === null) return false; + const { statusCode, body } = cause as { readonly statusCode?: unknown; readonly body?: unknown }; + if (statusCode !== 404 || typeof body !== "string") return false; + // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: classifying a third-party SDK's raw response body; a body that isn't JSON simply isn't this error + try { + // oxlint-disable-next-line executor/no-json-parse -- boundary: the autumn-js SDK hands back the response body as an unvalidated string + const parsed: unknown = JSON.parse(body); + if (typeof parsed !== "object" || parsed === null) return false; + return (parsed as { readonly code?: unknown }).code === CUSTOMER_NOT_FOUND_CODE; + } catch { + return false; + } +}; + // --------------------------------------------------------------------------- // Service interface // --------------------------------------------------------------------------- export type IAutumnService = Readonly<{ - use: (fn: (client: Autumn) => Promise) => Effect.Effect; + use: (fn: (client: Autumn) => Promise) => Effect.Effect; + /** + * Provision the organization's Autumn customer, creating it if Autumn has + * never seen it. Idempotent — safe to call on every org creation. + */ + ensureCustomer: (organizationId: string) => Effect.Effect; checkExecutionBalance: ( organizationId: string, - ) => Effect.Effect<{ readonly allowed: boolean }, AutumnError, never>; + ) => Effect.Effect<{ readonly allowed: boolean }, AutumnFailure, never>; /** * Fire-and-forget-safe execution usage tracker. Errors are caught and * logged; the returned Effect never fails. Callers typically @@ -48,6 +94,7 @@ const make = Effect.sync(() => { ); return { use: () => notConfigured, + ensureCustomer: () => notConfigured, checkExecutionBalance: () => notConfigured, trackExecution: () => Effect.void, } satisfies IAutumnService; @@ -62,16 +109,53 @@ const make = Effect.sync(() => { const use = (fn: (client: Autumn) => Promise) => Effect.tryPromise({ try: () => fn(client), - catch: (cause) => new AutumnError({ message: "Autumn SDK request failed", cause }), - }).pipe(Effect.withSpan(`autumn.${fn.name ?? "use"}`)); + catch: (cause): AutumnFailure => + isCustomerNotFoundCause(cause) + ? new AutumnCustomerNotFoundError({ + message: "Autumn has no customer for this organization", + cause, + }) + : new AutumnError({ message: "Autumn SDK request failed", cause }), + // An inline arrow's `name` is "" — not nullish — so `??` left every + // Autumn call tracing as the bare span `autumn.`. + }).pipe(Effect.withSpan(`autumn.${fn.name || "use"}`)); + + const ensureCustomer = (organizationId: string) => + Effect.asVoid(use((c) => c.customers.getOrCreate({ customerId: organizationId }))); + + /** + * Run `operation`; if Autumn answers "no such customer", provision the + * organization's customer and run it ONCE more. + * + * This is the seam that closes the provisioning hole. Both billing paths use + * non-creating endpoints, so an organization Autumn never learned about + * 404s here forever: the balance gate fails open (correct for an outage, + * catastrophic as a steady state) and every usage track is lost. Repairing + * the customer makes the retry land the call — and a genuine Autumn outage + * still fails with `AutumnError` and still pages, unretried. + */ + const withProvisionedCustomer = ( + organizationId: string, + operation: Effect.Effect, + ) => + operation.pipe( + Effect.catchTag("AutumnCustomerNotFoundError", () => + Effect.gen(function* () { + yield* Effect.annotateCurrentSpan({ "autumn.customer.provisioned": true }); + yield* ensureCustomer(organizationId); + return yield* operation; + }), + ), + ); const trackExecution = (organizationId: string) => Effect.gen(function* () { yield* Effect.annotateCurrentSpan({ "autumn.customer.id": organizationId }); - yield* use((c) => - c.track({ customerId: organizationId, featureId: "executions", value: 1 }), + yield* withProvisionedCustomer( + organizationId, + use((c) => c.track({ customerId: organizationId, featureId: "executions", value: 1 })), ).pipe( - Effect.catchTag("AutumnError", (error) => + Effect.catch((error) => Effect.gen(function* () { // Silent billing data loss is worth paging on: autumn.trackExecution // is fire-and-forget so the caller doesn't handle it themselves. @@ -88,13 +172,14 @@ const make = Effect.sync(() => { const checkExecutionBalance = (organizationId: string) => Effect.gen(function* () { yield* Effect.annotateCurrentSpan({ "autumn.customer.id": organizationId }); - const check = yield* use((c) => - c.check({ customerId: organizationId, featureId: "executions" }), + const check = yield* withProvisionedCustomer( + organizationId, + use((c) => c.check({ customerId: organizationId, featureId: "executions" })), ); return { allowed: check.allowed }; }).pipe(Effect.withSpan("autumn.checkExecutionBalance")); - return { use, checkExecutionBalance, trackExecution } satisfies IAutumnService; + return { use, ensureCustomer, checkExecutionBalance, trackExecution } satisfies IAutumnService; }); export class AutumnService extends Context.Service()( diff --git a/apps/cloud/src/mcp/agent-handler.ts b/apps/cloud/src/mcp/agent-handler.ts index 7738e79c6..0ec697c91 100644 --- a/apps/cloud/src/mcp/agent-handler.ts +++ b/apps/cloud/src/mcp/agent-handler.ts @@ -13,17 +13,26 @@ import { currentPropagationHeaders, readArtifactsEnabled, readElicitationMode, + readSearchToolsEnabled, withVerifiedIdentityHeaders, } from "@executor-js/cloudflare/mcp/do-headers"; import type { McpSessionProps } from "@executor-js/cloudflare/mcp/agent-durable-object"; +import { + classifyDurableObjectError, + durableObjectFailureResponse, + type DurableObjectFailure, +} from "@executor-js/cloudflare/mcp/durable-object-errors"; import { mcpSessionStub } from "@executor-js/cloudflare/mcp/session-stub"; import { wrapMcpSseResponse } from "../observability/memory-metrics"; import { WorkerTelemetryLive } from "../observability/telemetry"; import { cloudMcpAuth } from "./auth-provider"; +import { isMcpSessionMetaUnavailable } from "./session-meta"; import { McpSessionDOSqlite } from "./session-durable-object"; import { parseTraceparent } from "./traceparent"; +const MCP_SESSION_UNAVAILABLE_MESSAGE = "Session storage temporarily unavailable - please retry"; + const corsPreflightResponse = (): Response => new Response(null, { status: 204, @@ -79,6 +88,41 @@ const renderAuthError = ( }); }; +/** + * A Cloudflare *platform* Durable Object failure happened at one of this + * handler's stub touchpoints. Record what kind it was — on an exported span and + * in a structured log — so the production volume stays countable per cause + * (deploy reset vs storage timeout vs destroyed session) now that it is no + * longer a pile of 500s. + * + * Talking to a session DO means talking to a process the platform can reset out + * from under us: a deploy, a storage timeout, a backend blip, the session's own + * `ctx.abort("destroyed")`. None of those are application defects. An + * unrecognized failure never reaches here and keeps escaping as before. + */ +const recordDurableObjectFailure = ( + failure: DurableObjectFailure, + operation: string, +): Effect.Effect => + Effect.sync(() => { + console.warn( + JSON.stringify({ + event: "mcp_durable_object_platform_failure", + operation, + resetKind: failure.kind, + disposition: failure.disposition, + }), + ); + }).pipe( + Effect.withSpan("mcp.do.platform_failure", { + attributes: { + "mcp.do.reset_kind": failure.kind, + "mcp.do.reset_disposition": failure.disposition, + "mcp.do.reset_operation": operation, + }, + }), + ); + const authenticate = (request: Request) => Effect.gen(function* () { const auth = yield* McpAuthProvider; @@ -130,9 +174,17 @@ const propsForPrincipal = ( return { session: { organizationId: principal.organizationId, + // The org record the live membership check resolved microseconds ago, + // handed to the session DO so it never opens a connection of its own to + // re-read it. An unnamed org (no auth plane could resolve one) is + // omitted rather than sent empty, so the DO can tell "not carried" from + // "carried, and blank". + ...(principal.organizationName ? { organizationName: principal.organizationName } : {}), + ...(principal.organizationSlug ? { organizationSlug: principal.organizationSlug } : {}), userId: principal.accountId, elicitationMode: readElicitationMode(request), artifactsEnabled: readArtifactsEnabled(request), + searchToolsEnabled: readSearchToolsEnabled(request), resource, webOrigin: new URL(request.url).origin, }, @@ -199,10 +251,27 @@ export const makeCloudMcpAgentHandler = () => { } if (sessionId) { - const owner = await mcpSessionStub(env.MCP_SESSION, sessionId).validateMcpSessionOwner({ - accountId: outcome.principal.accountId, - organizationId: outcome.principal.organizationId, - }); + let owner: "ok" | "not_found" | "forbidden" | "terminated"; + // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary: a Durable Object stub RPC rejects with a plain platform Error, never a typed failure + try { + owner = await mcpSessionStub(env.MCP_SESSION, sessionId).validateMcpSessionOwner({ + accountId: outcome.principal.accountId, + organizationId: outcome.principal.organizationId, + }); + } catch (error) { + // The sibling stub touchpoints in this handler are both guarded — the + // `_cf_scheduleDestroy` call above with `Effect.ignore`, the + // `target.fetch` below with a catch — and this one was not, so a session + // whose DO had been destroyed or reset by the platform 500ed here before + // any of that handling could run. + const failure = classifyDurableObjectError(error); + if (!failure) { + // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary: an unrecognized failure is a real defect and must reach the runtime unchanged + throw error; + } + await runTraced(request, recordDurableObjectFailure(failure, "validate_session_owner")); + return durableObjectFailureResponse(failure); + } if (owner === "not_found") { return jsonRpcResponse(404, -32001, "Session not found"); } @@ -242,12 +311,35 @@ export const makeCloudMcpAgentHandler = () => { // DO ever getting to answer. Map it to the old envelope's reconnect // error for a dead session (e2e/cloud/mcp-protocol.test.ts expects the // client to be told to reconnect, matching a timed-out session). - // oxlint-disable-next-line executor/no-unknown-error-message -- adapter boundary: the abort reason is a plain runtime Error whose message IS the signal - if (Predicate.isError(error) && error.message === "destroyed") { - return jsonRpcResponse(404, -32001, "Session timed out, please reconnect"); + // + // The same catch now also covers the rest of the platform's reset + // vocabulary — a deploy, a storage timeout, a cancelled + // blockConcurrencyWhile — which reaches here through the agents SDK's own + // `getServerByName` retry and used to 500 identically. + // The session DO could not reach the organization directory to name the + // org (after its own bounded retry). Transient by construction, so it + // gets the same retryable envelope a WorkOS blip gets on the auth path — + // not an unclassified 500 the agents SDK then retries the whole DO + // operation over, which is what turned a 10s connect timeout into a + // half-minute client hang. + // + // Checked BEFORE the platform classifier: this is an application failure + // that merely escapes through the same seam, and it names its own cause. + // The classifier only recognizes the runtime's own reset vocabulary, so + // the two never contend — the order just keeps it that way if either + // vocabulary grows. + if (isMcpSessionMetaUnavailable(error)) { + return jsonRpcErrorBody(503, -32001, MCP_SESSION_UNAVAILABLE_MESSAGE, { + retryAfterSeconds: UNAVAILABLE_RETRY_AFTER_SECONDS, + }); + } + const failure = classifyDurableObjectError(error); + if (!failure) { + // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary: rethrow anything that isn't a recognized platform failure to the Workers runtime unchanged + throw error; } - // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary: rethrow anything that isn't the condemned-DO abort to the Workers runtime unchanged - throw error; + await runTraced(request, recordDurableObjectFailure(failure, "session_fetch")); + return durableObjectFailureResponse(failure); } // The agents SDK answers a bare DELETE with 204; the old envelope's // contract (see above) was 200 — rewrite for consistency. diff --git a/apps/cloud/src/mcp/auth-provider.test.ts b/apps/cloud/src/mcp/auth-provider.test.ts index 454573ade..bcd6009ef 100644 --- a/apps/cloud/src/mcp/auth-provider.test.ts +++ b/apps/cloud/src/mcp/auth-provider.test.ts @@ -70,9 +70,11 @@ const stubOrgAuthNoMembership = Layer.succeed(McpOrganizationAuth)({ authorize: () => Effect.succeed(null), }); -// `authorize` SUCCEEDS with an org id — active membership. +// `authorize` SUCCEEDS with the resolved org record — active membership. The +// record, not just the id: the session props carry the org's name and slug so +// the session DO never re-reads the row. const stubOrgAuthActive = Layer.succeed(McpOrganizationAuth)({ - authorize: () => Effect.succeed(ORG_ID), + authorize: () => Effect.succeed({ id: ORG_ID, name: "Stub Org", slug: "stub-org" }), }); // A failure that is not a WorkOSError at all (e.g. the per-request DB layer diff --git a/apps/cloud/src/mcp/auth-provider.ts b/apps/cloud/src/mcp/auth-provider.ts index 628007a2f..92384da16 100644 --- a/apps/cloud/src/mcp/auth-provider.ts +++ b/apps/cloud/src/mcp/auth-provider.ts @@ -21,8 +21,9 @@ // clearExistingSession. // - verified + org allowed -> Authenticated(principal) // -// The rich `mcp.request.annotate` client-fingerprint span (cloud-specific, no -// envelope seam) is emitted from here so telemetry parity is preserved. +// The rich client-fingerprint annotations (cloud-specific, no envelope seam) +// are stamped onto the `mcp.request` span from here so telemetry parity is +// preserved. // // The OAuth endpoints (/authorize, /token, /register) are NOT cloud's — they // live at WorkOS/AuthKit (external); only the two discovery docs are mounted. @@ -54,6 +55,7 @@ import { McpAuthLive, McpOrganizationAuth, McpOrganizationAuthLive, + type AuthorizedMcpOrganization, type McpAuthResult, type VerifiedToken, } from "./auth"; @@ -87,16 +89,24 @@ const ORGANIZATION_AUTHORIZE_UNAVAILABLE = /** * Enrich a cloud {@link VerifiedToken} (which carries only accountId + - * organizationId) into the full {@link Principal} the seam validates. The - * envelope only uses `accountId` + `organizationId` for ownership; cloud - * resolves org name/email inside the DO, so the cosmetic identity fields carry - * empty placeholders. `organizationId` is guaranteed non-null here because the - * Forbidden branch already rejected the no-org case before Authenticated. + * organizationId) into the full {@link Principal} the seam validates. + * + * The org name and slug come from the record the live membership check just + * resolved — this is the whole point of `authorize` returning the record rather + * than an id. They used to be dropped here (`organizationName: ""`), which left + * the session Durable Object to re-read the same row over a fresh database + * connection on every cold init. `email` stays a placeholder: the envelope only + * uses `accountId` + `organizationId` for ownership, and nothing downstream + * reads it. */ -const principalFromToken = (token: VerifiedToken, organizationId: string): Principal => ({ +const principalFromToken = ( + token: VerifiedToken, + organization: AuthorizedMcpOrganization, +): Principal => ({ accountId: token.accountId, - organizationId, - organizationName: "", + organizationId: organization.id, + organizationName: organization.name, + ...(organization.slug === undefined ? {} : { organizationSlug: organization.slug }), email: "", name: null, avatarUrl: null, @@ -222,9 +232,9 @@ export const cloudMcpAuthProviderLayer: Layer.Layer< // caller genuinely holds no active membership (revoked / never a member) // — a real Forbidden, which the handler may act on by condemning the // session. - const organizationId = authorizeResult.success; - if (!organizationId) return forbidden(NO_ORGANIZATION_MESSAGE, -32001); - return authenticated(principalFromToken(token, organizationId)); + const organization = authorizeResult.success; + if (!organization) return forbidden(NO_ORGANIZATION_MESSAGE, -32001); + return authenticated(principalFromToken(token, organization)); }); const toOutcome = (request: Request, result: McpAuthResult): Effect.Effect => { diff --git a/apps/cloud/src/mcp/auth.ts b/apps/cloud/src/mcp/auth.ts index 6ffdcf438..ee6bb8ca1 100644 --- a/apps/cloud/src/mcp/auth.ts +++ b/apps/cloud/src/mcp/auth.ts @@ -160,19 +160,32 @@ export class McpAuth extends Context.Service< } >()("@executor-js/cloud/McpAuth") {} +/** + * The organization an MCP request was authorized against. The full record, not + * just its id: the same request later needs the org's display name and slug to + * open a session, and re-reading the row for them (from the session Durable + * Object, on a fresh database connection) is a redundant failure point on a + * request that already has the answer. + */ +export type AuthorizedMcpOrganization = { + readonly id: string; + readonly name: string; + readonly slug?: string; +}; + export class McpOrganizationAuth extends Context.Service< McpOrganizationAuth, { /** * Authorize `accountId` against an org SELECTOR — a WorkOS org id * (`org_…`, from the token or a legacy URL) or the org's URL slug (the - * form the install card prints). Returns the resolved org id when the + * form the install card prints). Returns the resolved organization when the * caller holds an active membership, `null` otherwise. */ readonly authorize: ( accountId: string, organizationSelector: string, - ) => Effect.Effect; + ) => Effect.Effect; } >()("@executor-js/cloud/McpOrganizationAuth") {} @@ -204,7 +217,9 @@ const resolveOrgSelector = (selector: string) => ? Effect.succeed(selector) : Effect.gen(function* () { const users = yield* UserStoreService; - const org = yield* users.use((s) => s.getOrganizationBySlug(selector)); + const org = yield* users.use("getOrganizationBySlug", (s) => + s.getOrganizationBySlug(selector), + ); return org?.id ?? null; }); @@ -214,7 +229,9 @@ export const McpOrganizationAuthLive = Layer.succeed(McpOrganizationAuth)({ Effect.flatMap((organizationId) => organizationId ? authorizeOrganization(accountId, organizationId).pipe( - Effect.map((org) => (org ? org.id : null)), + Effect.map((org) => + org ? ({ id: org.id, name: org.name, slug: org.slug } as const) : null, + ), ) : Effect.succeed(null), ), diff --git a/apps/cloud/src/mcp/session-durable-object.ts b/apps/cloud/src/mcp/session-durable-object.ts index 31d6d4bf6..5ff5ee2f0 100644 --- a/apps/cloud/src/mcp/session-durable-object.ts +++ b/apps/cloud/src/mcp/session-durable-object.ts @@ -58,7 +58,7 @@ import { buildExecuteDescription, type ResumeResponse } from "@executor-js/execu // `SessionAuthLive` instead.) import { CoreSharedServices } from "../auth/workos"; import { UserStoreService } from "../auth/context"; -import { resolveOrganization } from "../auth/organization"; +import { resolveSessionMetaForToken } from "./session-meta"; import { DbService, combinedSchema, @@ -66,14 +66,12 @@ import { type DrizzleDb, type DbServiceShape, } from "../db/db"; -import { makeExecutionStack } from "../engine/execution-stack"; -import { preloadQuickJs } from "../quickjs"; -import { CloudMeteredExecutionStackLayer } from "../engine/execution-stack-metered"; import { AutumnService } from "../extensions/billing/service"; import { DoTelemetryLive, flushTracerProvider } from "../observability/telemetry"; import { captureCause as reportCause, captureCauseEffect as reportCauseEffect, + claimCauseHandledByDurableObject, tagCurrentSentryScopeWithCurrentOtelSpan, } from "../observability"; import { parseTraceparent } from "./traceparent"; @@ -109,10 +107,6 @@ type CloudSessionDbHandle = DbServiceShape & { readonly end: () => Promise; }; -class OrganizationNotFoundError extends Data.TaggedError("OrganizationNotFoundError")<{ - readonly organizationId: string; -}> {} - class McpModelResumeForwardError extends Data.TaggedError("McpModelResumeForwardError")<{ readonly cause: unknown; }> {} @@ -211,27 +205,27 @@ export class McpSessionDOSqlite extends McpAgentSessionDOBase { + protected override resolveSessionMeta( + token: McpSessionInit, + storedMeta: SessionMeta | null, + ): Effect.Effect { + // The database handle is opened LAZILY: on the props and stored paths — the + // overwhelming majority of inits — nothing here touches Postgres at all, + // which is the whole point. postgres.js only dials on first query, so + // building the handle costs nothing; `ensuring` still closes it. const dbHandle = makeEphemeralDb(); - return Effect.gen(function* () { - const org = yield* resolveOrganization(token.organizationId); - if (!org) { - return yield* new OrganizationNotFoundError({ organizationId: token.organizationId }); - } - return { - organizationId: org.id, - organizationName: org.name, - organizationSlug: org.slug, - userId: token.userId, - resource: token.resource, - elicitationMode: token.elicitationMode, - artifactsEnabled: token.artifactsEnabled, - } satisfies SessionMeta; - }).pipe( + return resolveSessionMetaForToken(token, storedMeta).pipe( Effect.withSpan("McpSessionDOSqlite.resolveSessionMeta"), Effect.provide(makeSessionServices(dbHandle)), Effect.ensuring(Effect.promise(() => dbHandle.end())), - // oxlint-disable-next-line executor/no-effect-escape-hatch -- boundary: a vanished org is a defect; the worker already verified the bearer + // The base's `resolveSessionMeta` seam has no error channel, and a + // Durable Object's `init` can only reject its Promise — so a failure has + // to leave as a defect. What changed is WHAT leaves: an unreachable + // organization directory is now a bounded, classified + // `McpSessionMetaUnavailableError` whose message the worker recognises + // and renders as a retryable 503 (see `agent-handler.ts`), instead of an + // unclassified Postgres cause that produced a 500 and a client hang. + // oxlint-disable-next-line executor/no-effect-escape-hatch -- boundary: the DO init seam is Promise-only; the failure is classified before it dies Effect.orDie, ); } @@ -242,6 +236,31 @@ export class McpSessionDOSqlite extends McpAgentSessionDOBase { const self = this; return Effect.gen(function* () { + // Imported here rather than at module scope. Cloudflare requires a + // Durable Object class to be exported from the Worker entry, so every + // static import this module makes is evaluated by *every* cold isolate — + // including the ones that only render a page or forward a passthrough + // proxy and never open an MCP session. These three roots pull the whole + // code-execution stack (sucrase, ajv, QuickJS-WASM): measured at 1.9 MB + // of the Worker's startup closure, for code only a real session runs. + // `apps/cloud/scripts/start-closure.mjs` reports that number and will + // show it moving back if these become static again. + const [{ preloadQuickJs }, { makeExecutionStack }, { CloudMeteredExecutionStackLayer }] = + yield* Effect.promise( + () => + Promise.all([ + import("../quickjs"), + import("../engine/execution-stack"), + import("../engine/execution-stack-metered"), + ]) as Promise< + [ + typeof import("../quickjs"), + typeof import("../engine/execution-stack"), + typeof import("../engine/execution-stack-metered"), + ] + >, + ); + // QuickJS-WASM must be loaded before anything asks for a sandbox: the // default variant cannot fetch its own `.wasm` on Workers. Cloud runs // user `execute` code on the dynamic-worker runtime, but the artifact @@ -280,6 +299,9 @@ export class McpSessionDOSqlite extends McpAgentSessionDOBase): Effect.Effect { + return claimCauseHandledByDurableObject; + } + // Best-effort export the DO isolate's buffered spans after the RPC settles, // so a dying init/handleRequest can ship its own spans (and the exception + // stack recorded on them) — not just the worker-side `mcp.do.*` span. Keep it diff --git a/apps/cloud/src/mcp/session-meta.node.test.ts b/apps/cloud/src/mcp/session-meta.node.test.ts new file mode 100644 index 000000000..d1dcdac23 --- /dev/null +++ b/apps/cloud/src/mcp/session-meta.node.test.ts @@ -0,0 +1,167 @@ +// Where an MCP session's organization identity comes from, and what happens +// when the only remaining source — the database — is unreachable. +// +// The production defect: every cold session-DO init read the `organizations` +// row over a brand-new Postgres connection, even though the worker had resolved +// that exact row microseconds earlier on the same request, and even when the DO +// already held the answer in its own storage. A connect timeout on that +// unnecessary connection became an unclassified defect and killed `initialize`. +import { describe, expect, it } from "@effect/vitest"; +import { Effect, Layer, Predicate, Result } from "effect"; + +import { defaultMcpResource } from "@executor-js/host-mcp"; +import type { McpSessionInit, SessionMeta } from "@executor-js/cloudflare/mcp/agent-durable-object"; + +import { UserStoreService } from "../auth/context"; +import { UserStoreError } from "../auth/errors"; +import { WorkOSClient, type WorkOSClientService } from "../auth/workos"; +import { + isMcpSessionMetaUnavailable, + resolveSessionMetaForToken, + SESSION_META_DB_RETRIES, +} from "./session-meta"; + +const TOKEN: McpSessionInit = { + organizationId: "org_test", + userId: "user_test", + elicitationMode: "model", + resource: defaultMcpResource, + artifactsEnabled: true, +}; + +const STORED: SessionMeta = { + organizationId: "org_test", + organizationName: "Stored Org", + organizationSlug: "stored-org", + userId: "user_test", + resource: defaultMcpResource, +}; + +/** A user store that always fails the way a wedged Hyperdrive endpoint does, + * counting how many times it was asked. */ +const countingConnectTimeoutStore = (): { + readonly layer: Layer.Layer; + readonly calls: () => number; +} => { + let calls = 0; + return { + calls: () => calls, + layer: Layer.succeed(UserStoreService)({ + use: (operation: string) => + Effect.suspend(() => { + calls += 1; + return Effect.fail(new UserStoreError({ operation, reason: "connect_timeout" })); + }), + } as UserStoreService["Service"]), + }; +}; + +const namingStore = (): { + readonly layer: Layer.Layer; + readonly calls: () => number; +} => { + let calls = 0; + return { + calls: () => calls, + layer: Layer.succeed(UserStoreService)({ + use: (_operation: string, fn: (store: never) => Promise) => + Effect.suspend(() => { + calls += 1; + return Effect.promise(() => + fn({ + getOrganization: async (id: string) => ({ + id, + name: "Database Org", + slug: "database-org", + }), + } as never), + ); + }), + } as UserStoreService["Service"]), + }; +}; + +const unusedWorkOS = Layer.succeed( + WorkOSClient, + new Proxy({} as WorkOSClientService, { + get: (_target, prop) => () => Effect.die(`unexpected WorkOSClient.${String(prop)} call`), + }), +); + +// The props source — the one the overwhelming majority of inits take — is +// covered black-box by `e2e/cloud/mcp-session-cold-init.test.ts`. What stays +// here is what that scenario cannot reach: the sources it falls back to, and +// what an unreachable database does to them. +describe("resolveSessionMetaForToken", () => { + it("falls back to the meta this session already stored, without touching the store", async () => { + const store = countingConnectTimeoutStore(); + + const meta = await Effect.runPromise( + resolveSessionMetaForToken(TOKEN, STORED).pipe( + Effect.provide(Layer.mergeAll(store.layer, unusedWorkOS)), + ), + ); + + expect(meta.organizationName).toBe("Stored Org"); + expect(meta.organizationSlug).toBe("stored-org"); + expect(store.calls(), "a restore reuses what it persisted").toBe(0); + }); + + it("reads the database only when nothing else names the org", async () => { + const store = namingStore(); + + const meta = await Effect.runPromise( + resolveSessionMetaForToken(TOKEN, null).pipe( + Effect.provide(Layer.mergeAll(store.layer, unusedWorkOS)), + ), + ); + + expect(meta.organizationName).toBe("Database Org"); + expect(store.calls()).toBe(1); + }); + + it("retries a connect timeout a bounded number of times, then fails retryably", async () => { + const store = countingConnectTimeoutStore(); + + const result = await Effect.runPromise( + resolveSessionMetaForToken(TOKEN, null).pipe( + Effect.provide(Layer.mergeAll(store.layer, unusedWorkOS)), + Effect.result, + ), + ); + + expect(Result.isFailure(result), "an unreachable directory is a failure, not a defect").toBe( + true, + ); + if (!Result.isFailure(result)) return; + expect(Predicate.isTagged(result.failure, "McpSessionMetaUnavailableError")).toBe(true); + expect( + isMcpSessionMetaUnavailable(result.failure), + "the worker can recognise it across the Durable Object boundary", + ).toBe(true); + expect(store.calls(), "bounded: the first attempt plus its retries").toBe( + SESSION_META_DB_RETRIES + 1, + ); + }); + + it("does not retry a deterministic query failure", async () => { + let calls = 0; + const store = Layer.succeed(UserStoreService)({ + use: (operation: string) => + Effect.suspend(() => { + calls += 1; + return Effect.fail(new UserStoreError({ operation, reason: "query" })); + }), + } as UserStoreService["Service"]); + + const result = await Effect.runPromise( + resolveSessionMetaForToken(TOKEN, null).pipe( + Effect.provide(Layer.mergeAll(store, unusedWorkOS)), + Effect.result, + ), + ); + + expect(Result.isFailure(result)).toBe(true); + expect(calls, "a query the server answered will answer the same way again").toBe(1); + }); +}); diff --git a/apps/cloud/src/mcp/session-meta.ts b/apps/cloud/src/mcp/session-meta.ts new file mode 100644 index 000000000..a0bd1df81 --- /dev/null +++ b/apps/cloud/src/mcp/session-meta.ts @@ -0,0 +1,209 @@ +// --------------------------------------------------------------------------- +// Where an MCP session's organization identity comes from. +// +// Three sources, in the order they are preferred: +// +// props — the org record the worker resolved while authorizing THIS +// request, carried in the session props. Free, and always the +// freshest answer available. +// stored — the meta this session Durable Object already persisted for the +// same organization on an earlier init. Free, and correct for a +// session that already exists. +// database — an actual read of the `organizations` row. +// +// Only the third can fail, and it is the one that used to run unconditionally. +// Every cold DO init opened a brand-new Postgres connection through Hyperdrive +// purely to re-read a row the worker had loaded microseconds earlier on the +// same request and then discarded; when that connection could not be +// established the read hung for the full connect budget and the failure was +// turned into a defect, killing `initialize` on a database that was, at that +// same moment, answering the worker's own queries in milliseconds. +// +// So: prefer what the request already knows, and when the database genuinely is +// the only source, give it a bounded retry and a CLASSIFIED failure — the same +// transient-vs-definitive split the WorkOS membership check already uses in +// `auth-provider.ts` — instead of an unclassified defect. +// --------------------------------------------------------------------------- + +import { Data, Effect, Predicate, Result, Schedule } from "effect"; + +import type { McpSessionInit, SessionMeta } from "@executor-js/cloudflare/mcp/agent-durable-object"; + +import { UserStoreService } from "../auth/context"; +import { WorkOSClient } from "../auth/workos"; +import { + isDefinitiveWorkOSDenial, + isTransientUserStoreReason, + type UserStoreError, +} from "../auth/errors"; +import { resolveOrganization } from "../auth/organization"; + +export type SessionMetaSource = "props" | "stored" | "database"; + +export const SESSION_META_SOURCE_ATTRIBUTE = "mcp.session.meta_source"; + +/** + * The wire marker for "the organization directory was unreachable, try again". + * + * A Durable Object's `init` can only reject its Promise, so this travels to the + * worker as an ordinary error message across the DO boundary — the same + * mechanism the condemned-session `"destroyed"` abort already uses. The worker + * matches on it to answer a retryable 503 instead of letting an unclassified + * 500 (and the agents SDK's own DO-operation retry on top of it) turn a + * transient blip into a half-minute client hang. + */ +export const MCP_SESSION_META_UNAVAILABLE = "mcp_session_meta_unavailable"; + +export class McpSessionMetaUnavailableError extends Data.TaggedError( + "McpSessionMetaUnavailableError", +)<{ + readonly reason: string; + readonly attempts: number; +}> { + override get message(): string { + return `${MCP_SESSION_META_UNAVAILABLE}: organization directory unavailable (${this.reason}) after ${this.attempts} attempts`; + } +} + +export class OrganizationNotFoundError extends Data.TaggedError("OrganizationNotFoundError")<{ + readonly organizationId: string; +}> {} + +/** Does this failure, seen at the worker, mean "the session DO could not reach + * the organization directory"? Matches on the message because that is what + * survives the Durable Object RPC boundary. */ +export const isMcpSessionMetaUnavailable = (error: unknown): boolean => + // oxlint-disable-next-line executor/no-unknown-error-message -- adapter boundary: a Durable Object rejection reaches the worker as a plain Error whose message IS the signal (same mechanism as the "destroyed" abort) + Predicate.isError(error) && error.message.includes(MCP_SESSION_META_UNAVAILABLE); + +/** + * Retries for the database path. Deliberately small: the point is to ride out a + * single bad connection attempt, not to sit on a client's `initialize` while a + * database stays down. Exhausting them answers the client quickly and + * retryably, which is strictly better than hanging. + */ +export const SESSION_META_DB_RETRIES = 2; + +const RETRY_SCHEDULE = Schedule.both( + Schedule.exponential("200 millis"), + Schedule.recurs(SESSION_META_DB_RETRIES), +); + +const isUserStoreError = Predicate.isTagged("UserStoreError") as ( + error: unknown, +) => error is UserStoreError; + +/** + * Is this org-lookup failure worth another attempt? A connection that never + * opened is; a query the server answered with an error, or a WorkOS denial, is + * not — retrying those only burns the client's init budget. + */ +export const isRetryableOrganizationLookupFailure = (failure: unknown): boolean => { + if (isUserStoreError(failure)) return isTransientUserStoreReason(failure.reason); + if (isDefinitiveWorkOSDenial(failure)) return false; + // A WorkOS blip (429/5xx/timeout/network) — the same class the MCP auth path + // already treats as retryable. + return Predicate.isTagged(failure, "WorkOSError"); +}; + +const failureReason = (failure: unknown): string => + isUserStoreError(failure) ? failure.reason : "upstream"; + +const metaFromIdentity = ( + token: McpSessionInit, + organization: { readonly name: string; readonly slug?: string }, +): SessionMeta => ({ + organizationId: token.organizationId, + organizationName: organization.name, + ...(organization.slug === undefined ? {} : { organizationSlug: organization.slug }), + userId: token.userId, + resource: token.resource, + elicitationMode: token.elicitationMode, + artifactsEnabled: token.artifactsEnabled, + searchToolsEnabled: token.searchToolsEnabled, +}); + +/** + * Read the organization row, retrying only failures a retry can clear, and + * surfacing an exhausted retry as a typed, retryable failure rather than a + * defect. + */ +const organizationFromDatabase = ( + organizationId: string, +): Effect.Effect< + { readonly id: string; readonly name: string; readonly slug?: string }, + McpSessionMetaUnavailableError | OrganizationNotFoundError, + UserStoreService | WorkOSClient +> => + Effect.gen(function* () { + let attempts = 0; + // `Effect.retry` retries every failure it sees, so definitive failures are + // lifted OUT of the error channel before the schedule ever runs and only + // the retryable ones are left in it. + const attempt = Effect.suspend(() => { + attempts += 1; + return resolveOrganization(organizationId).pipe( + Effect.result, + Effect.flatMap((outcome) => + Result.isFailure(outcome) && isRetryableOrganizationLookupFailure(outcome.failure) + ? Effect.fail(outcome.failure) + : Effect.succeed(outcome), + ), + ); + }); + + // Two nested Results: the outer one is "the retries ran out", the inner one + // is "the lookup failed definitively on the first look". Both mean the same + // thing to the caller. + const retried = yield* attempt.pipe(Effect.retry(RETRY_SCHEDULE), Effect.result); + const outcome = Result.isFailure(retried) ? Result.fail(retried.failure) : retried.success; + + if (Result.isFailure(outcome)) { + return yield* new McpSessionMetaUnavailableError({ + reason: failureReason(outcome.failure), + attempts, + }); + } + const organization = outcome.success; + if (!organization) return yield* new OrganizationNotFoundError({ organizationId }); + return organization; + }).pipe( + Effect.withSpan("mcp.session.resolve_organization", { + attributes: { "mcp.auth.organization_id": organizationId }, + }), + ); + +/** + * Build the session meta for an init, preferring the org identity the request + * already carries over any read of the organization directory. + * + * `storedMeta` is this DO's own persisted meta for the SAME organization (the + * base Durable Object only offers a matching one), or `null`. + */ +export const resolveSessionMetaForToken = ( + token: McpSessionInit, + storedMeta: SessionMeta | null, +): Effect.Effect< + SessionMeta, + McpSessionMetaUnavailableError | OrganizationNotFoundError, + UserStoreService | WorkOSClient +> => + Effect.gen(function* () { + const fromProps = token.organizationName; + if (fromProps) { + yield* Effect.annotateCurrentSpan(SESSION_META_SOURCE_ATTRIBUTE, "props"); + return metaFromIdentity(token, { name: fromProps, slug: token.organizationSlug }); + } + + if (storedMeta?.organizationName) { + yield* Effect.annotateCurrentSpan(SESSION_META_SOURCE_ATTRIBUTE, "stored"); + return metaFromIdentity(token, { + name: storedMeta.organizationName, + slug: storedMeta.organizationSlug, + }); + } + + yield* Effect.annotateCurrentSpan(SESSION_META_SOURCE_ATTRIBUTE, "database"); + const organization = yield* organizationFromDatabase(token.organizationId); + return metaFromIdentity(token, organization); + }); diff --git a/apps/cloud/src/mcp/telemetry.test.ts b/apps/cloud/src/mcp/telemetry.test.ts index 173241f99..df150bc9e 100644 --- a/apps/cloud/src/mcp/telemetry.test.ts +++ b/apps/cloud/src/mcp/telemetry.test.ts @@ -103,3 +103,58 @@ describe("annotateMcpRequest — cancellation join keys", () => { }).pipe(Effect.withSpan("mcp.request"), Effect.withTracer(tracer)); }); }); + +describe("annotateMcpRequest — executed-code capture", () => { + it.effect("stamps the script on execute calls", () => { + const { tracer, attributesOf } = makeRecordingTracer(); + return Effect.gen(function* () { + yield* annotate( + postRequest({ + jsonrpc: "2.0", + id: 8, + method: "tools/call", + params: { name: "execute", arguments: { code: "return 6 * 7;" } }, + }), + ); + const attributes = attributesOf("mcp.request"); + expectDefined(attributes); + expect(attributes.get("mcp.execute.code")).toBe("return 6 * 7;"); + }).pipe(Effect.withSpan("mcp.request"), Effect.withTracer(tracer)); + }); + + it.effect("caps oversized scripts with a truncation marker", () => { + const { tracer, attributesOf } = makeRecordingTracer(); + return Effect.gen(function* () { + const code = "x".repeat(12_000); + yield* annotate( + postRequest({ + jsonrpc: "2.0", + id: 9, + method: "tools/call", + params: { name: "execute", arguments: { code } }, + }), + ); + const attributes = attributesOf("mcp.request"); + expectDefined(attributes); + const captured = attributes.get("mcp.execute.code"); + expect(captured).toBe(`${"x".repeat(10_000)}\n… [truncated 2000 chars]`); + }).pipe(Effect.withSpan("mcp.request"), Effect.withTracer(tracer)); + }); + + it.effect("does not capture arguments of other tools", () => { + const { tracer, attributesOf } = makeRecordingTracer(); + return Effect.gen(function* () { + yield* annotate( + postRequest({ + jsonrpc: "2.0", + id: 10, + method: "tools/call", + params: { name: "resume", arguments: { code: "not an execute call" } }, + }), + ); + const attributes = attributesOf("mcp.request"); + expectDefined(attributes); + expect(attributes.get("mcp.execute.code")).toBeUndefined(); + }).pipe(Effect.withSpan("mcp.request"), Effect.withTracer(tracer)); + }); +}); diff --git a/apps/cloud/src/mcp/telemetry.ts b/apps/cloud/src/mcp/telemetry.ts index 5f4c370f7..3f0c30025 100644 --- a/apps/cloud/src/mcp/telemetry.ts +++ b/apps/cloud/src/mcp/telemetry.ts @@ -102,7 +102,10 @@ const InitializeParams = Schema.Struct({ capabilities: Schema.optional(UnknownRecord), }); -const NamedParams = Schema.Struct({ name: Schema.optional(Schema.String) }); +const NamedParams = Schema.Struct({ + name: Schema.optional(Schema.String), + arguments: Schema.optional(UnknownRecord), +}); const UriParams = Schema.Struct({ uri: Schema.optional(Schema.String) }); // `notifications/cancelled` carries no JSON-RPC id of its own, but its params @@ -134,6 +137,29 @@ const readJsonRpcEnvelope = (request: Request): Effect.Effect | undefined, +): Record => { + if (name !== "execute" && name !== "execute-action") return {}; + const code = args?.["code"]; + if (typeof code !== "string") return {}; + return { + "mcp.execute.code": + code.length > MAX_CODE_ATTR_CHARS + ? `${code.slice(0, MAX_CODE_ATTR_CHARS)}\n… [truncated ${code.length - MAX_CODE_ATTR_CHARS} chars]` + : code, + }; +}; + const methodAttrs = (envelope: JsonRpcEnvelope): Record => { const params = envelope.params ?? {}; return Match.value(envelope.method).pipe( @@ -154,7 +180,10 @@ const methodAttrs = (envelope: JsonRpcEnvelope): Record => { Match.when("tools/call", () => Option.match(decodeNamedParams(params), { onNone: () => ({}) as Record, - onSome: ({ name }) => (name ? { "mcp.tool.name": name } : {}), + onSome: ({ name, arguments: args }) => ({ + ...(name ? { "mcp.tool.name": name } : {}), + ...executeCodeAttrs(name, args), + }), }), ), Match.whenOr("resources/read", "resources/subscribe", () => @@ -237,5 +266,4 @@ export const annotateMcpRequest = ( }; yield* Effect.annotateCurrentSpan(attrs); - yield* Effect.annotateCurrentSpan(attrs).pipe(Effect.withSpan("mcp.request.annotate")); }); diff --git a/apps/cloud/src/mcp/traceparent.ts b/apps/cloud/src/mcp/traceparent.ts index 0be604025..72555df4c 100644 --- a/apps/cloud/src/mcp/traceparent.ts +++ b/apps/cloud/src/mcp/traceparent.ts @@ -1,13 +1,13 @@ // --------------------------------------------------------------------------- // W3C traceparent parsing shared by the worker edge (server.ts), the MCP agent -// handler, and the session DO. Single-sourced so the producer (the worker span -// stamping traceparent onto the forwarded request) and the consumers (the -// Effect programs joining that span) cannot drift on the header grammar. +// handler, and the session DO. The header grammar itself is single-sourced in +// `@executor-js/cloudflare/mcp/do-headers` beside the producer that stamps the +// header; this module only layers the OTel `tracestate` carrier on top for the +// consumers that join the span via the OTel API. // --------------------------------------------------------------------------- import { createTraceState } from "@opentelemetry/api"; - -const TRACEPARENT_PATTERN = /^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/; +import { parseTraceparentHeader } from "@executor-js/cloudflare/mcp/do-headers"; export type IncomingSpanContext = { readonly traceId: string; @@ -20,13 +20,10 @@ export const parseTraceparent = ( traceparent: string | null | undefined, tracestate: string | null | undefined, ): IncomingSpanContext | null => { - if (!traceparent) return null; - const match = TRACEPARENT_PATTERN.exec(traceparent); - if (!match) return null; + const parsed = parseTraceparentHeader(traceparent); + if (!parsed) return null; return { - traceId: match[2]!, - spanId: match[3]!, - traceFlags: parseInt(match[4]!, 16), + ...parsed, ...(tracestate ? { traceState: createTraceState(tracestate) } : {}), }; }; diff --git a/apps/cloud/src/observability/header-redaction.ts b/apps/cloud/src/observability/header-redaction.ts new file mode 100644 index 000000000..8c7fd3fc0 --- /dev/null +++ b/apps/cloud/src/observability/header-redaction.ts @@ -0,0 +1,19 @@ +// --------------------------------------------------------------------------- +// Span header redaction for fibers composed under the cloud telemetry layers +// (the Effect HTTP server tracer and any request-scoped client use). The +// allowlist itself lives beside the hosted HTTP client in +// `@executor-js/sdk/host-internal`, which also binds it directly to that +// client: consumers capture the client value at construction, so a context +// layer alone cannot reach their request fibers. This layer is the +// server-side/defense-in-depth half of the same decision; see +// hosted-http-client.ts for the rationale and the allowlist. +// --------------------------------------------------------------------------- + +import { Layer } from "effect"; +import { Headers } from "effect/unstable/http"; +import { spanRedactedHeaderNames } from "@executor-js/sdk/host-internal"; + +export const SpanHeaderRedactionLive: Layer.Layer = Layer.succeed( + Headers.CurrentRedactedNames, + spanRedactedHeaderNames, +); diff --git a/apps/cloud/src/observability/index.ts b/apps/cloud/src/observability/index.ts index 6d1cd7e99..2ad2755a9 100644 --- a/apps/cloud/src/observability/index.ts +++ b/apps/cloud/src/observability/index.ts @@ -12,10 +12,11 @@ import * as Sentry from "@sentry/cloudflare"; import type { ErrorEvent, Scope } from "@sentry/cloudflare"; -import { Cause, Effect, Layer } from "effect"; +import { Cause, Effect, Layer, Predicate } from "effect"; import type * as Tracer from "effect/Tracer"; import { ErrorCapture } from "@executor-js/api"; +import { withStableGroupingFingerprint } from "@executor-js/sdk/sentry-grouping"; // Drizzle/postgres-js include the failing SQL (params + bound values) in // their error message. For OpenAPI source inserts that's 1MB+ of spec @@ -30,6 +31,21 @@ export const OTEL_TRACE_ID_TAG = "otel_trace_id"; export const OTEL_SPAN_ID_TAG = "otel_span_id"; export const SENTRY_EVENT_ID_ATTRIBUTE = "sentry.event_id"; +/** + * Set by the MCP session Durable Object when it has finished deciding what to + * do about a cause — reported it, or classified it as an expected Cloudflare + * platform reset and deliberately not reported it. + * + * `instrumentDurableObjectWithSentry` wraps the DO's entry points and captures + * the same rejection again as it escapes, which is why one platform reset + * opened two issues for the same event. The DO is the better owner — it has + * the session, the org, the OTEL correlation and the classification — so its + * claim wins and the auto-instrumentation's echo is dropped in `beforeSend`. + * Nothing the DO does not claim is affected. + */ +export const DO_CAUSE_OWNER_TAG = "mcp.do.cause_owner"; +export const DO_CAUSE_OWNER_VALUE = "durable_object"; + export type OtelCorrelationContext = { readonly traceId: string; readonly spanId: string; @@ -99,10 +115,25 @@ export const tagCurrentSentryScopeWithCurrentOtelSpan: Effect.Effect { + if (event.tags?.[DO_CAUSE_OWNER_TAG] !== DO_CAUSE_OWNER_VALUE) return false; + const mechanism = event.exception?.values?.[0]?.mechanism?.type; + return typeof mechanism === "string" && mechanism.startsWith("auto."); +}; + export const beforeSendWithOtelCorrelation = ( event: ErrorEvent, options?: { readonly logPayload?: boolean }, -): ErrorEvent => { +): ErrorEvent | null => { + if (isClaimedDurableObjectEcho(event)) return null; if (options?.logPayload) { console.info( JSON.stringify({ @@ -116,6 +147,53 @@ export const beforeSendWithOtelCorrelation = ( return event; }; +/** + * The single `beforeSend` the worker and its Durable Objects install. + * + * The worker ships as content-hashed chunks and its frames are not resolved + * back to source, so Sentry's default grouping keys on names like + * `execution-rate-limit-` and re-opens every issue on the next deploy. + * `withStableGroupingFingerprint` pins a key with the hash normalized out; + * events with no hashed grouping input are left on the default algorithm. + * + * The two stages are independent and compose in this order: the capture-owner + * pass decides WHETHER the event is reported at all (a cause the Durable + * Object already claimed is dropped, and a dropped event is never + * fingerprinted), and the grouping pass then decides HOW whatever survives is + * grouped. + */ +export const beforeSendCloudEvent = ( + event: ErrorEvent, + options?: { readonly logPayload?: boolean }, +): ErrorEvent | null => { + const reported = beforeSendWithOtelCorrelation(event, options); + return reported === null ? null : withStableGroupingFingerprint(reported); +}; + +/** + * The Sentry options the worker and every Durable Object install. It lives + * beside the `beforeSend` it wires so the composition is covered by + * observability.test.ts; `server.ts` only passes this through. + * + * NOTE: do NOT enable `instrumentPrototypeMethods`. It walks the DO prototype + * and reads every property — including accessors — to find methods to wrap, + * which invokes the `sessionId` getter with `this` bound to the prototype + * (where `ctx` is undefined) and throws during construction, 500ing every + * session create / cold restore. The DO captures its own errors via the + * `captureCause` seam (→ Sentry) instead. + */ +export const cloudSentryOptions = (env: Env) => ({ + dsn: env.SENTRY_DSN, + tracesSampleRate: 0, + enableLogs: true, + sendDefaultPii: true, + skipOpenTelemetrySetup: true, + beforeSend: (event: ErrorEvent) => + beforeSendCloudEvent(event, { + logPayload: !env.SENTRY_DSN || env.SENTRY_OTEL_LOG_PAYLOAD === "true", + }), +}); + export const addCurrentOtelCorrelationTags = < T extends { readonly tags?: Record }, >( @@ -144,14 +222,75 @@ export const sentryPayloadForCause = ( return { primary: input, pretty: null }; }; +// Safe classification fields our tagged errors carry (`UserStoreError.operation` +// / `.reason`, `WorkOSError.status`). They are promoted to Sentry TAGS because +// the pretty cause is only an `extra`, and Sentry's server-side scrubber +// replaces that extra with "[Filtered]" — leaving an issue with no failing +// operation and no reason in it at all. Tags survive, group, and are +// searchable. Values are failure modes and operation names; never a query, a +// value, or anything customer-derived. +const CLASSIFICATION_TAG_FIELDS = ["operation", "reason", "status"] as const; + +/** The errors those fields are read from. An allowlist, because the fields are + * only known to be safe on the errors this app defines. */ +const CLASSIFIED_ERROR_TAGS = [ + "UserStoreError", + "WorkOSError", + "McpSessionMetaUnavailableError", +] as const; + +const MAX_CLASSIFICATION_TAG_CHARS = 120; + +const MAX_CAUSE_NESTING = 3; + +/** Every error value a cause carries, failures and defects alike. A defect can + * itself be a `Cause` (an inner `runPromise` rejecting with its own squashed + * cause), so the walk unwraps a few levels. */ +const errorValuesOf = (input: unknown, depth = 0): readonly unknown[] => { + if (depth >= MAX_CAUSE_NESTING) return []; + if (!Cause.isCause(input)) return [input]; + const values: unknown[] = []; + for (const reason of input.reasons) { + if (Cause.isFailReason(reason)) values.push(reason.error); + else if (Cause.isDieReason(reason)) values.push(...errorValuesOf(reason.defect, depth + 1)); + } + return values; +}; + +/** Read the classification fields off the tagged errors inside a cause. First + * writer wins, so the innermost reported error names the issue. */ +const classificationTagsOf = (input: unknown): Readonly> => { + const tags: Record = {}; + for (const candidate of errorValuesOf(input)) { + if (typeof candidate !== "object" || candidate === null) continue; + // Only the errors whose fields we know are safe classifications. Without + // this, any foreign object in the cause that happens to carry an + // `operation` / `reason` / `status` field would have that value promoted + // onto a tag, and a foreign field is not known to be free of a query, a + // value, or anything customer-derived. + if (!CLASSIFIED_ERROR_TAGS.some((tag) => Predicate.isTagged(candidate, tag))) continue; + const tagged = candidate as Record; + for (const field of CLASSIFICATION_TAG_FIELDS) { + const value = tagged[field]; + if (tags[field] !== undefined) continue; + if (typeof value === "string" || typeof value === "number") { + tags[field] = String(value).slice(0, MAX_CLASSIFICATION_TAG_CHARS); + } + } + } + return tags; +}; + export const captureCause = ( input: unknown, context: OtelCorrelationContext | null = null, ): string | undefined => { const { primary, pretty } = sentryPayloadForCause(input); + const classification = classificationTagsOf(input); tagCurrentSentryScopeWithOtelContext(context); return Sentry.captureException(primary, (scope) => { tagSentryScopeWithOtelContext(scope, context); + for (const [key, value] of Object.entries(classification)) scope.setTag(key, value); if (pretty !== null) scope.setExtra("cause", pretty); return scope; }); @@ -167,6 +306,15 @@ export const captureCauseEffect = (input: unknown): Effect.Effect = Effect.sync(() => { + Sentry.getCurrentScope().setTag(DO_CAUSE_OWNER_TAG, DO_CAUSE_OWNER_VALUE); +}); + export const ErrorCaptureLive: Layer.Layer = Layer.succeed( ErrorCapture, ErrorCapture.of({ diff --git a/apps/cloud/src/observability/observability.test.ts b/apps/cloud/src/observability/observability.test.ts index b4f69a8d8..a75a8e617 100644 --- a/apps/cloud/src/observability/observability.test.ts +++ b/apps/cloud/src/observability/observability.test.ts @@ -2,8 +2,15 @@ import { describe, expect, it } from "@effect/vitest"; import { Cause, Effect } from "effect"; import type * as Tracer from "effect/Tracer"; +import type { ErrorEvent } from "@sentry/cloudflare"; + import { addCurrentOtelCorrelationTags, + beforeSendCloudEvent, + beforeSendWithOtelCorrelation, + cloudSentryOptions, + DO_CAUSE_OWNER_TAG, + DO_CAUSE_OWNER_VALUE, OTEL_SPAN_ID_TAG, OTEL_TRACE_ID_TAG, sentryPayloadForCause, @@ -79,6 +86,83 @@ describe("sentryPayloadForCause", () => { }); }); +// Grouping keys are decided inside the Sentry SDK and never appear on any +// product surface, so the e2e harness cannot observe them. The split is: +// e2e/cloud/sentry-otel-correlation.test.ts proves the worker really installs +// `cloudSentryOptions.beforeSend` (its correlation payload only exists if that +// hook ran), and the tests here prove the hook it installs fingerprints. +describe("Sentry grouping", () => { + // The worker bundle ships as content-hashed chunks, so the only module name + // Sentry ever sees for a given frame changes on every deploy. + const workerEvent = (chunkHash: string): ErrorEvent => ({ + type: undefined, + exception: { + values: [ + { + type: "GateCheckTimeoutError", + value: "balance check timed out", + stacktrace: { + frames: [ + { + filename: `/assets/execution-rate-limit-${chunkHash}.js`, + module: `execution-rate-limit-${chunkHash}`, + function: "timeoutOrElse", + in_app: true, + }, + ], + }, + }, + ], + }, + }); + + it("pins one fingerprint across two deploys of the same chunk", () => { + const before = beforeSendCloudEvent(workerEvent("BAuwphPA"), {}); + const after = beforeSendCloudEvent(workerEvent("DkcPBbWe"), {}); + + expect(before?.fingerprint).toBeDefined(); + expect(before?.fingerprint).toEqual(after?.fingerprint); + }); + + it("leaves unhashed events on Sentry's default grouping", () => { + const event: ErrorEvent = { + type: undefined, + exception: { + values: [ + { + type: "AutumnError", + stacktrace: { + frames: [ + { filename: "/src/engine/execution-gate.ts", function: "checkExecutionBalance" }, + ], + }, + }, + ], + }, + }; + + const sent = beforeSendCloudEvent(event, {}); + + expect(sent).not.toBeNull(); + expect(sent?.fingerprint).toBeUndefined(); + }); + + // The wiring check: this is the exact object handed to `Sentry.withSentry` + // and `instrumentDurableObjectWithSentry` in server.ts. If the normalizer is + // ever dropped from the hook the worker installs, this fails. + it("the options the worker and DOs install carry the fingerprinting hook", () => { + const options = cloudSentryOptions({ SENTRY_DSN: "https://public@example.invalid/1" } as Env); + const sent = options.beforeSend(workerEvent("BAuwphPA")); + + expect(sent?.fingerprint).toEqual([ + "GateCheckTimeoutError", + "timeoutOrElse@execution-rate-limit", + ]); + // Same source, next deploy, new chunk hash — one issue, not two. + expect(options.beforeSend(workerEvent("DkcPBbWe"))?.fingerprint).toEqual(sent?.fingerprint); + }); +}); + describe("Sentry OTel correlation", () => { it.effect("adds tags from the active Effect span", () => Effect.gen(function* () { @@ -91,3 +175,128 @@ describe("Sentry OTel correlation", () => { }).pipe(Effect.withSpan("test.sentry_capture"), Effect.withTracer(makeFixedTracer())), ); }); + +// One Durable Object failure used to open two Sentry issues: the DO's own +// `captureCause` seam reported it (mechanism `generic`), and then +// `instrumentDurableObjectWithSentry` reported the very same rejection again as +// it escaped the method (mechanism `auto.faas.cloudflare.durable_object`). The +// DO is the owner — it has the session, the classification and the OTel +// correlation — so its claim suppresses the echo and nothing else. +describe("Durable Object capture ownership", () => { + const doEcho = (overrides: Partial = {}): ErrorEvent => ({ + type: undefined, + tags: { [DO_CAUSE_OWNER_TAG]: DO_CAUSE_OWNER_VALUE }, + exception: { + values: [ + { + type: "Error", + value: "Durable Object reset because its code was updated.", + mechanism: { type: "auto.faas.cloudflare.durable_object", handled: false }, + }, + ], + }, + ...overrides, + }); + + it("drops the auto-instrumentation's copy of a cause the DO already claimed", () => { + expect(beforeSendWithOtelCorrelation(doEcho())).toBeNull(); + }); + + it("keeps the DO's own report, which carries no auto mechanism", () => { + const own = doEcho({ + exception: { + values: [ + { + type: "Error", + value: "Durable Object reset because its code was updated.", + mechanism: { type: "generic", handled: true }, + }, + ], + }, + }); + expect(beforeSendWithOtelCorrelation(own)).not.toBeNull(); + }); + + // An alarm crash or a transport fault is never claimed by the DO seam, and + // the auto-instrumentation is the ONLY thing that reports it. Dropping those + // would trade duplicate noise for silence. + it("keeps an unclaimed Durable Object failure", () => { + const unclaimed = doEcho({ tags: {} }); + expect(beforeSendWithOtelCorrelation(unclaimed)).not.toBeNull(); + }); + + it("keeps ordinary worker events untouched", () => { + const workerEvent: ErrorEvent = { + type: undefined, + tags: { [OTEL_TRACE_ID_TAG]: traceId }, + exception: { + values: [ + { + type: "TypeError", + value: "x is not a function", + mechanism: { type: "auto.http.cloudflare", handled: false }, + }, + ], + }, + }; + expect(beforeSendWithOtelCorrelation(workerEvent)).not.toBeNull(); + }); + + // The two stages of the installed `beforeSend` answer different questions and + // must both keep working: capture ownership decides WHETHER an event is + // reported, stable grouping decides HOW a reported one is grouped. A dropped + // event is never fingerprinted, and a surviving one still is. + describe("composed with stable grouping", () => { + const hashedFrames = (chunkHash: string) => ({ + stacktrace: { + frames: [ + { + filename: `/assets/session-durable-object-${chunkHash}.js`, + module: `session-durable-object-${chunkHash}`, + function: "handleSessionRequest", + in_app: true, + }, + ], + }, + }); + + it("drops a claimed echo rather than fingerprinting it", () => { + const echo = doEcho({ + exception: { + values: [ + { + type: "Error", + value: "Durable Object reset because its code was updated.", + mechanism: { type: "auto.faas.cloudflare.durable_object", handled: false }, + ...hashedFrames("BAuwphPA"), + }, + ], + }, + }); + + expect(beforeSendCloudEvent(echo, {})).toBeNull(); + }); + + it("pins a stable fingerprint on the report the DO itself owns", () => { + const ownReport = (chunkHash: string): ErrorEvent => + doEcho({ + exception: { + values: [ + { + type: "Error", + value: "Durable Object reset because its code was updated.", + mechanism: { type: "generic", handled: true }, + ...hashedFrames(chunkHash), + }, + ], + }, + }); + + const before = beforeSendCloudEvent(ownReport("BAuwphPA"), {}); + const after = beforeSendCloudEvent(ownReport("DkcPBbWe"), {}); + + expect(before?.fingerprint).toBeDefined(); + expect(before?.fingerprint).toEqual(after?.fingerprint); + }); + }); +}); diff --git a/apps/cloud/src/observability/telemetry.ts b/apps/cloud/src/observability/telemetry.ts index 7bd5baa58..3e992a215 100644 --- a/apps/cloud/src/observability/telemetry.ts +++ b/apps/cloud/src/observability/telemetry.ts @@ -45,6 +45,7 @@ import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from "@opentelemetry/semantic import { env } from "cloudflare:workers"; import { Effect, Layer } from "effect"; +import { SpanHeaderRedactionLive } from "./header-redaction"; import { CountingSpanExporter, CountingSpanProcessor, @@ -53,7 +54,36 @@ import { } from "./memory-metrics"; const SERVICE_NAME = "executor-cloud"; -const SERVICE_VERSION = "1.0.0"; + +// `service.version` is the Cloudflare Worker version id (from the +// `version_metadata` binding) so any span links back to the exact deploy in a +// step-change investigation; "dev" is the documented default for hosts without +// the binding (local dev, older test workers). `executor.commit_sha` rides +// along when CI passed it (`wrangler deploy --var GIT_COMMIT_SHA:...`). +const serviceVersion = (): string => env.CF_VERSION_METADATA?.id ?? "dev"; + +// One id per isolate: distinguishes "many isolates each paying a cold cost" +// from "one isolate is slow", and makes per-isolate cache behavior (JWKS, +// module caches) measurable from Axiom. The Aug 2026 latency investigation +// stalled for lack of exactly this attribute. +// +// Generated LAZILY on first use, not at module scope: workerd forbids random +// generation (and I/O) in global scope and Cloudflare's upload validation +// rejects the whole deploy for it (error 10021). First use is inside +// `installTracerProvider()` / the telemetry layer build, which both run in a +// request handler, so the id is still one-per-isolate. +let isolateInstanceId: string | null = null; +let isolateStartedAt: number | null = null; + +const resourceAttributes = (): Record => { + isolateInstanceId ??= crypto.randomUUID(); + isolateStartedAt ??= Date.now(); + return { + "service.instance.id": isolateInstanceId, + "executor.isolate_started_at": new Date(isolateStartedAt).toISOString(), + ...(env.GIT_COMMIT_SHA === undefined ? {} : { "executor.commit_sha": env.GIT_COMMIT_SHA }), + }; +}; // Module-scope: one provider per isolate, never shut down. The provider holds // the SimpleSpanProcessor + OTLP exporter, so any tracer reference captured by @@ -65,7 +95,8 @@ const ensureGlobalTracerProvider = (): boolean => { provider = new WebTracerProvider({ resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: SERVICE_NAME, - [ATTR_SERVICE_VERSION]: SERVICE_VERSION, + [ATTR_SERVICE_VERSION]: serviceVersion(), + ...resourceAttributes(), }), spanProcessors: (() => { let countingProcessor: CountingSpanProcessor; @@ -124,15 +155,25 @@ export const flushTracerProvider = async (): Promise => { }; const makeTelemetryLive = (): Layer.Layer => - Layer.unwrap( - Effect.sync(() => - ensureGlobalTracerProvider() - ? OtelTracer.layerGlobal.pipe( - Layer.provide( - Resource.layer({ serviceName: SERVICE_NAME, serviceVersion: SERVICE_VERSION }), - ), - ) - : Layer.empty, + Layer.mergeAll( + // Redaction applies even when the exporter is not installed: Effect still + // builds spans (and their header attributes) in-memory, and any future + // consumer of those spans must never observe an unredacted credential. + SpanHeaderRedactionLive, + Layer.unwrap( + Effect.sync(() => + ensureGlobalTracerProvider() + ? OtelTracer.layerGlobal.pipe( + Layer.provide( + Resource.layer({ + serviceName: SERVICE_NAME, + serviceVersion: serviceVersion(), + attributes: resourceAttributes(), + }), + ), + ) + : Layer.empty, + ), ), ); diff --git a/apps/cloud/src/routes/__root.tsx b/apps/cloud/src/routes/__root.tsx index 474eb8045..517d53b0b 100644 --- a/apps/cloud/src/routes/__root.tsx +++ b/apps/cloud/src/routes/__root.tsx @@ -23,7 +23,6 @@ import { Toaster } from "@executor-js/react/components/sonner"; import { ExecutorPluginsProvider } from "@executor-js/sdk/client"; import { ArtifactRendererProvider } from "@executor-js/react/api/artifact-renderer"; import { plugins as clientPlugins } from "virtual:executor/plugins-client"; -import type { AuthHint } from "@executor-js/react/multiplayer/auth-hint"; import { AuthProvider, useAuth } from "../web/auth"; import { loginPath } from "../auth/return-to"; import { ONBOARDING_PATHS, PUBLIC_PATHS } from "../auth/route-paths"; @@ -111,27 +110,6 @@ function NotFoundPage() { export const Route = createRootRoute({ notFoundComponent: NotFoundPage, - // What the SSR gate attached to this document request (ssr-gate.ts → - // middleware context → serverContext). Loader data is dehydrated, so the - // client's first render sees the SAME values the server rendered with — the - // two can't disagree: - // - authHint: the verified identity, seeding AuthProvider's initial state. - // - origin: the request origin, seeding the server connection so the - // connect-card MCP URL SSRs as the real origin instead of the - // 127.0.0.1 client-side default (which would flash to the real - // value at hydration). - // Client-side re-runs have no serverContext and return null; both consumers - // fall back gracefully (the hint is already held, the origin to the - // window-derived global). - loader: (opts) => { - const serverContext = ( - opts as { serverContext?: { authHint?: AuthHint | null; origin?: string } } - ).serverContext; - return { - authHint: serverContext?.authHint ?? null, - origin: serverContext?.origin ?? null, - }; - }, head: () => ({ meta: [ { charSet: "utf-8" }, @@ -171,12 +149,15 @@ function RootDocument({ children }: { children: React.ReactNode }) { } function RootComponent() { - const { authHint, origin } = Route.useLoaderData(); + // SPA mode: no per-request server render, so nothing is dehydrated. Auth + // seeds from the client-readable hint cookie one frame after mount + // (AuthProvider's own fallback), and origin-derived UI reads the + // window-derived global. return ( - - + + @@ -213,7 +194,7 @@ function ShellErrorFallback() { ); } -function AuthGate({ ssrOrigin }: { ssrOrigin: string | null }) { +function AuthGate() { const auth = useAuth(); const location = useLocation(); const navigate = useNavigate(); @@ -265,11 +246,11 @@ function AuthGate({ ssrOrigin }: { ssrOrigin: string | null }) { } // Every state that isn't "authenticated with an org, on a page that wants - // the shell" is a moment between redirects or an edge the gates make - // near-impossible (a verified user whose hint hasn't seeded yet). Neutral + // the shell" is a moment between redirects, or the one frame between mount + // and the hint cookie seeding (SPA mode reads it in an effect). Neutral // blank — the one placeholder that's correct whatever happens next. The // app-shell skeleton this file used to render here is exactly the - // wrong-UI flash the SSR gate + hint exist to prevent. + // wrong-UI flash the document gate + hint exist to prevent. if (auth.status === "loading" || auth.status === "unauthenticated") { return ; } @@ -286,12 +267,6 @@ function AuthGate({ ssrOrigin }: { ssrOrigin: string | null }) { return urlOrgSlug ? : ; } - // Seed the server connection from the SSR origin so origin-derived UI (the - // connect card's MCP URL) renders the real host on the first paint instead - // of the 127.0.0.1 default the client-side global falls back to during SSR. - // Null on client loader re-runs → undefined → the window-derived global, - // which is the same origin, so the key never changes and nothing remounts. - const connection = ssrOrigin ? ({ kind: "http", origin: ssrOrigin } as const) : undefined; const activeSlug = auth.organization.slug; // The org context's slug feeds the connect card's `//mcp` install URL. // Prefer the URL's slug over the session's: on first paint `auth.organization` @@ -315,11 +290,7 @@ function AuthGate({ ssrOrigin }: { ssrOrigin: string | null }) { canonicalization remounts the registry so anything fetched header-less on first paint (rejected server-side) is refetched with the org header. */} - + }> void) const [finalizingPlan, setFinalizingPlan] = useState(null); const plansRef = useRef(plans); plansRef.current = plans; - + const refetchRef = useRef(refetch); + refetchRef.current = refetch; + const armedAtRef = useRef(0); + + // One-shot: move the URL marker into state. Only this step is single-shot — + // the poll below keys off the STATE, because effects do not live as long as + // state: the shell re-runs effects shortly after mount (auth seeds a frame + // after first paint and the suspense boundary re-reveals), and when the poll + // lived inside this effect its re-run saw the already-stripped URL, so the + // cleanup killed the interval and nothing ever refetched — the page sat on + // "Activating" forever while the webhook had long landed. useEffect(() => { const params = new URLSearchParams(window.location.search); const attachedPlanId = params.get(CHECKOUT_RETURN_PARAM); @@ -56,25 +66,34 @@ function useRefreshAfterCheckout(plans: Plan[] | undefined, refetch: () => void) params.delete(CHECKOUT_RETURN_PARAM); const query = params.toString(); window.history.replaceState({}, "", `${window.location.pathname}${query ? `?${query}` : ""}`); + armedAtRef.current = Date.now(); setFinalizingPlan(attachedPlanId); + }, []); + + // Poll while a plan is finalizing. Re-armable: if the effect is recycled the + // state still says which plan is settling, so the interval comes right back. + // The timeout is measured from when the marker was consumed, not from the + // current effect run, so recycles cannot extend it. refetch goes through a + // ref so the interval always calls the CURRENT client's refetch, even after + // the billing provider rebuilds. + useEffect(() => { + if (!finalizingPlan) return; const reflected = () => - plansRef.current?.find((p) => p.id === attachedPlanId)?.customerEligibility?.status === + plansRef.current?.find((p) => p.id === finalizingPlan)?.customerEligibility?.status === "active"; - let elapsed = 0; - refetch(); + refetchRef.current(); const interval = setInterval(() => { - elapsed += 1500; - if (reflected() || elapsed >= 20_000) { + if (reflected() || Date.now() - armedAtRef.current >= 20_000) { clearInterval(interval); setFinalizingPlan(null); return; } - refetch(); + refetchRef.current(); }, 1500); return () => clearInterval(interval); - }, [refetch]); + }, [finalizingPlan]); // Drop the optimistic state the moment the refetched data reflects the plan, // so it does not linger until the next poll tick after the webhook lands. @@ -103,7 +122,7 @@ const PLAN_META: Record, // and bounces already-signed-in visitors straight back to it. export const Route = createFileRoute("/login")({ diff --git a/apps/cloud/src/server.ts b/apps/cloud/src/server.ts index 12aac6773..4f5bf7628 100644 --- a/apps/cloud/src/server.ts +++ b/apps/cloud/src/server.ts @@ -1,6 +1,5 @@ import { DurableObject } from "cloudflare:workers"; -import { SpanKind, SpanStatusCode, context, trace } from "@opentelemetry/api"; -import type { ErrorEvent } from "@sentry/cloudflare"; +import { SpanKind, SpanStatusCode, context, trace, type SpanContext } from "@opentelemetry/api"; import { ATTR_HTTP_REQUEST_METHOD, ATTR_HTTP_RESPONSE_STATUS_CODE, @@ -11,13 +10,15 @@ import { import * as Sentry from "@sentry/cloudflare"; import handler from "@tanstack/react-start/server-entry"; -import { isAppOwnedPath } from "./app-paths"; +import { isAppOwnedPath, servedByAppPlane } from "./app-paths"; +import { marketingProxyRequest } from "./edge/marketing"; +import { passthroughResponse } from "./edge/passthrough"; import { makeCloudMcpAgentHandler } from "./mcp/agent-handler"; import { classifyMcpPath, prepareMcpOrgScope } from "./mcp/mount"; import { parseTraceparent } from "./mcp/traceparent"; import { McpSessionDOSqlite as McpSessionDOBase } from "./mcp/session-durable-object"; import { - beforeSendWithOtelCorrelation, + cloudSentryOptions, captureCause, otelCorrelationContextFromOpenTelemetrySpan, SENTRY_EVENT_ID_ATTRIBUTE, @@ -26,28 +27,6 @@ import { import { browserTracesResponse } from "./observability/browser-traces"; import { flushTracerProvider, installTracerProvider } from "./observability/telemetry"; -// --------------------------------------------------------------------------- -// Sentry config -// --------------------------------------------------------------------------- - -const sentryOptions = (env: Env) => ({ - dsn: env.SENTRY_DSN, - tracesSampleRate: 0, - enableLogs: true, - sendDefaultPii: true, - skipOpenTelemetrySetup: true, - beforeSend: (event: ErrorEvent) => - beforeSendWithOtelCorrelation(event, { - logPayload: !env.SENTRY_DSN || env.SENTRY_OTEL_LOG_PAYLOAD === "true", - }), - // NOTE: do NOT enable `instrumentPrototypeMethods`. It walks the DO prototype - // and reads every property — including accessors — to find methods to wrap, - // which invokes the `sessionId` getter with `this` bound to the prototype - // (where `ctx` is undefined) and throws during construction, 500ing every - // session create / cold restore. The DO captures its own errors via the - // `captureCause` seam (→ Sentry) instead. -}); - // --------------------------------------------------------------------------- // Durable Object — wrapped with Sentry so DO errors land in Sentry (inits the // client inside the DO isolate, which plain `Sentry.captureException` cannot @@ -56,7 +35,7 @@ const sentryOptions = (env: Env) => ({ // --------------------------------------------------------------------------- export const McpSessionDOSqlite = Sentry.instrumentDurableObjectWithSentry( - sentryOptions, + cloudSentryOptions, McpSessionDOBase, ); @@ -85,15 +64,12 @@ export { McpExecutionOwnerDirectoryDO } from "@executor-js/cloudflare/mcp/execut // migration — without the OTel-SDK version-conflict that package would now // drag in (it pins `@opentelemetry/otlp-* ^0.200.0`, we ship ^0.214.0). // -// ONLY for paths the Effect app does not own. App-owned paths (/api/*, /mcp, -// /.well-known/* — see app-paths.ts) get their `http.server` span from -// Effect's own HttpMiddleware.tracer, which parses `traceparent` itself and -// parents the workos/store/db child spans. Wrapping those here too produced -// two identical sibling `http.server` spans per request (scope -// `executor-cloud-worker` next to scope `executor-cloud`) — double ingest, -// and the waterfall showed a childless twin. The worker span remains for -// everything Effect never sees: Start SSR, the marketing proxy, /_astro -// assets. +// App-owned paths (/api/* and /.well-known/* — see app-paths.ts) get their +// `http.server` span from Effect's HttpMiddleware tracer. `/mcp` is dispatched +// directly and uses `traceCloudMcpRequest` below so the agent handler can skip +// the entire span envelope for negative-cache hits. Other paths keep this +// worker span. Wrapping Effect-owned paths here too produced duplicate sibling +// spans per request. // // SimpleSpanProcessor exports synchronously at span end but the underlying // `fetch()` to Axiom is fire-and-forget; the Worker may terminate before it @@ -101,17 +77,208 @@ export { McpExecutionOwnerDirectoryDO } from "@executor-js/cloudflare/mcp/execut // until the in-flight export resolves. // --------------------------------------------------------------------------- -const fetchHandler = handler.fetch as ( +const rawFetchHandler = handler.fetch as ( request: Request, env: Env, ctx: ExecutionContext, ) => Response | Promise; +/** + * Every entry into TanStack Start goes through here so `startGraphEntered` + * reflects whether this isolate has already paid the lazy `loadEntries` + * import — the cost that dominates a cold page request. + */ +const fetchHandler = ( + request: Request, + env: Env, + ctx: ExecutionContext, +): Response | Promise => { + markStartGraphEntered(); + return rawFetchHandler(request, env, ctx); +}; + const tracer = trace.getTracer("executor-cloud-worker"); + +const traceparentValueFor = (spanContext: SpanContext): string => + `00-${spanContext.traceId}-${spanContext.spanId}-${(spanContext.traceFlags & 0xff).toString(16).padStart(2, "0")}`; + +const withTraceparent = (request: Request, spanContext: SpanContext): Request => { + const headers = new Headers(request.headers); + headers.set("traceparent", traceparentValueFor(spanContext)); + return new Request(request, { headers }); +}; + +const traceCloudMcpRequest = async ( + request: Request, + _env: Env, + ctx: ExecutionContext, + handle: (tracedRequest: Request) => Promise, +): Promise => { + if (!installTracerProvider()) return handle(request); + + const url = new URL(request.url); + const inbound = parseTraceparent(request.headers.get("traceparent"), null); + const parentContext = inbound + ? trace.setSpanContext(context.active(), { + traceId: inbound.traceId, + spanId: inbound.spanId, + traceFlags: inbound.traceFlags, + isRemote: true, + }) + : context.active(); + + return tracer.startActiveSpan( + `http.server ${request.method}`, + { kind: SpanKind.SERVER }, + parentContext, + async (span) => { + span.setAttribute(ATTR_HTTP_REQUEST_METHOD, request.method); + span.setAttribute(ATTR_URL_FULL, request.url); + span.setAttribute(ATTR_URL_PATH, url.pathname); + span.setAttribute(ATTR_URL_SCHEME, url.protocol.replace(/:$/, "")); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; observe response/error for span status, keep trace export alive after the Agents bridge resolves or rejects + try { + const response = await handle(withTraceparent(request, span.spanContext())); + span.setAttribute(ATTR_HTTP_RESPONSE_STATUS_CODE, response.status); + if (response.status >= 500) { + span.setStatus({ code: SpanStatusCode.ERROR, message: `HTTP ${response.status}` }); + } + return response; + } catch (err) { + // oxlint-disable-next-line executor/no-instanceof-error, executor/no-unknown-error-message -- adapter boundary: Cloudflare's fetch callback throws untyped; normalized only for the OTel span record, the original error is rethrown below + const cause = err instanceof Error ? err : String(err); + span.recordException(cause); + // oxlint-disable-next-line executor/no-unknown-error-message -- adapter boundary: same normalization as the recordException line above + const message = typeof cause === "string" ? cause : cause.message; + span.setStatus({ code: SpanStatusCode.ERROR, message }); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; preserve original error to Cloudflare runtime + throw err; + } finally { + span.end(); + ctx.waitUntil(flushTracerProvider()); + } + }, + ); +}; + const mcpAgentHandler = makeCloudMcpAgentHandler(); +// --------------------------------------------------------------------------- +// Isolate lifecycle signals +// --------------------------------------------------------------------------- +// +// The Aug 2026 page-latency hunt kept stalling on one blind spot: every span we +// emit starts INSIDE the fetch handler, so nothing could distinguish "this +// isolate is cold and paying Start's lazy `loadEntries` import" from "this +// isolate came up slowly before our code ran at all". Both look like one slow +// span. These three attributes make the distinction queryable: +// +// executor.isolate.request_seq - 1 means this request is the isolate's +// first; page requests measured 1.04 per +// isolate, which is why nearly every one +// pays the cold cost. +// executor.start_graph.entered - whether anything had already driven +// TanStack Start in this isolate. /mcp and +// the passthrough proxies return before +// `fetchHandler`, so an isolate can serve +// plenty of traffic and still be cold here. +// Measured with a temporary entry-lag probe correlated against `wrangler tail` +// event timestamps: a cold isolate (`request_seq` 1) spends **840-1844ms** +// coming up before our first line runs, while a reused one spends **0-2ms**. +// That startup is invisible to every span we emit, and it sits on top of the +// ~3.1s `loadEntries` import — together the 3-5s a signed-in page costs. +// +// executor.isolate.id - identifies the isolate itself, so reuse can +// be counted directly instead of inferred. +// executor.isolate.age_ms - ms since this isolate served its first +// request. +// +// The last two exist because the Aug 2026 hunt inferred "isolates stopped being +// reused" from a latency cutoff (requests slower than 1s were called cold) and +// then built a size-based theory on top of that proxy. The theory was wrong: +// reverting the offending packages restored production while moving the +// evaluated module closure by 0.02 MB (see scripts/start-closure.mjs). Grouping +// by isolate id answers "how many requests did this isolate serve, and were the +// slow ones its first?" directly, which no latency threshold can. +// +// All of it is cheap: two increments, one lazy uuid, and no I/O. +let isolateRequestSeq = 0; +let startGraphEntered = false; +// Minted on first request rather than at module scope: Workers reject random +// number generation during global-scope evaluation. +let isolateId: string | undefined; +let isolateFirstSeenAt = 0; + +const identifyIsolate = (): { readonly id: string; readonly ageMs: number } => { + if (isolateId === undefined) { + isolateId = crypto.randomUUID(); + isolateFirstSeenAt = Date.now(); + } + return { id: isolateId, ageMs: Date.now() - isolateFirstSeenAt }; +}; + +const markStartGraphEntered = (): void => { + startGraphEntered = true; +}; + +// --------------------------------------------------------------------------- +// Serving `/api/*` without entering TanStack Start. +// --------------------------------------------------------------------------- +// +// Everything under `/api` is the Effect app (`ExecutorApp.make`'s web handler) +// and uses no part of the router, React, or SSR. But it was dispatched from a +// Start *request middleware*, so reaching it meant paying Start's lazy +// `loadEntries` import of the whole server graph first. Measured on production +// 2026-08-19, splitting `/api/*` by whether the isolate had already loaded that +// graph: warm p50 **186ms**, cold p50 **2129ms**, with 28% of API requests cold. +// The dashboard fires many `/api/*` calls in parallel and waits for the slowest, +// so that cold tail is what the app actually feels like. +// +// So `/api` joins marketing, `/docs`, the PostHog proxy and `/mcp` at the Worker +// entry: classify and dispatch before anything touches Start. The evaluated +// closure for an API request drops from the full Start graph to the Worker's own +// (see `scripts/start-closure.mjs`). +// +// `servedByAppPlane` (./app-paths) decides which paths qualify — two under +// `/api` are claimed by Start's middleware first and must keep their old route. + +// Instantiated on the first request that needs it and memoized per isolate, +// mirroring `start.ts`'s `getApp`. The import stays dynamic so an isolate that +// only serves pages or proxies never evaluates the app graph at all. +let appPlane: ReturnType | undefined; +let appGraphEntered = false; + +const getAppPlane = async (): Promise> => { + if (appPlane === undefined) { + const { cloudApiHandler } = await import("./app"); + appPlane = cloudApiHandler(); + appGraphEntered = true; + } + return appPlane; +}; + const cloudflareHandler: ExportedHandler = { fetch: async (request, env, ctx) => { + isolateRequestSeq += 1; + + // Public pages must not enter TanStack Start: its first-request dynamic + // import loads the entire React + Effect server graph and can take seconds + // on a cold isolate. Classify and service-bind marketing at the Worker + // entry, before telemetry or fetchHandler touches that graph. + const marketingRequest = marketingProxyRequest(request); + const marketing: Fetcher | undefined = env.MARKETING; + if (marketingRequest && marketing) return marketing.fetch(marketingRequest); + + // Same reasoning, same seam: `/docs` and the PostHog proxy forward to an + // external origin and never touch the router, React, or the Effect app. + // Left in Start's middleware they still paid its lazy `loadEntries` import + // first — measured at p50 3.1s on a cold isolate, against p50 33ms for the + // request's own work, on a Worker where 1,666 dispatches spread across + // 1,608 isolates (so nearly every request is cold). Forward before Start. + const passthroughPath = new URL(request.url).pathname; + const passthrough = passthroughResponse(request, passthroughPath); + if (passthrough) return passthrough; + // Browser OTLP ingress — before the server span opens: exporter traffic // must never trace itself (the browser already excludes /v1/traces from // its own tracing for the same reason). @@ -120,10 +287,24 @@ const cloudflareHandler: ExportedHandler = { // The MCP dispatch is classified up front, independent of whether // telemetry installs — an unset `AXIOM_TOKEN` (tracer not installed) must // never take /mcp requests down with it. See `installTracerProvider`'s - // early return below: it only governs the tracing envelope for - // non-MCP paths. + // early return below: the handler invokes it for uncached MCP traffic, and + // this entry invokes it for non-MCP paths. const url = new URL(request.url); const mcpRoute = classifyMcpPath(url.pathname); + if (mcpRoute?.kind === "mcp") { + // The Cloudflare Agents MCP bridge needs the platform ExecutionContext + // to pass authenticated session props into the hibernatable DO. + // Discovery docs still flow through the app-level MCP envelope. + const forwarded = prepareMcpOrgScope(request); + // /mcp leaves the Effect app for the Agents bridge, so no downstream + // HttpMiddleware.tracer opens the request envelope — this worker span is + // THE `http.server` span for MCP traffic, and its context is stamped onto + // the forwarded traceparent so the agent handler and session DO parent + // under it instead of exporting orphaned roots. + return traceCloudMcpRequest(forwarded, env, ctx, (tracedRequest) => + Promise.resolve(mcpAgentHandler(tracedRequest, env, ctx)), + ); + } const tracingInstalled = installTracerProvider(); // Join the caller's W3C trace when the request carries one — the web UI // sends traceparent on every API fetch, so the browser's spans and this @@ -138,69 +319,63 @@ const cloudflareHandler: ExportedHandler = { isRemote: true, }) : context.active(); - if (mcpRoute?.kind === "mcp") { - // The Cloudflare Agents MCP bridge needs the platform ExecutionContext - // to pass authenticated session props into the hibernatable DO. - // Discovery docs still flow through the app-level MCP envelope. - const forwarded = prepareMcpOrgScope(request); - if (!tracingInstalled) { - return mcpAgentHandler(forwarded, env, ctx); - } - // /mcp left the Effect app in the Agents-bridge migration, so no - // downstream HttpMiddleware.tracer opens the request envelope anymore — - // this worker span is now THE `http.server` span for MCP traffic. Its - // context is stamped onto the forwarded request's traceparent so the - // agent handler's Effect programs (mcp.request and children) and the - // session DO parent under it instead of exporting orphaned roots. + if (!tracingInstalled) { + return fetchHandler(request, env, ctx); + } + // Effect-served paths bring their own http.server span (with traceparent + // join) — a second SERVER span here would duplicate it (the header note). + // What they do NOT cover is the time between this invocation starting and + // the Effect router opening its span (Start dispatch, middleware, lazy + // module graph): during the Aug 2026 regression that gap was seconds of + // invisible wall time. `worker.dispatch` is an INTERNAL parent that + // brackets the whole invocation; Effect's http.server span joins under it + // via the injected traceparent, so gap = dispatch minus server span. + if (isAppOwnedPath(url.pathname)) { return tracer.startActiveSpan( - `http.server ${request.method}`, - { kind: SpanKind.SERVER }, + "worker.dispatch", + { kind: SpanKind.INTERNAL }, parentContext, async (span) => { span.setAttribute(ATTR_HTTP_REQUEST_METHOD, request.method); - span.setAttribute(ATTR_URL_FULL, request.url); span.setAttribute(ATTR_URL_PATH, url.pathname); - span.setAttribute(ATTR_URL_SCHEME, url.protocol.replace(/:$/, "")); - const spanContext = span.spanContext(); - const headers = new Headers(forwarded.headers); - headers.set( - "traceparent", - `00-${spanContext.traceId}-${spanContext.spanId}-${(spanContext.traceFlags & 0xff).toString(16).padStart(2, "0")}`, - ); - // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; observe response/error for span status, keep trace export alive after the Agents bridge resolves or rejects + const isolate = identifyIsolate(); + span.setAttribute("executor.isolate.request_seq", isolateRequestSeq); + span.setAttribute("executor.start_graph.entered", startGraphEntered); + span.setAttribute("executor.isolate.id", isolate.id); + span.setAttribute("executor.isolate.age_ms", isolate.ageMs); + // Which plane served this: "app" skipped the Start graph entirely, so + // `start_graph.entered` says nothing about it. `app_graph.entered` + // is the app-plane analogue - false means this request paid for the + // Effect graph's first evaluation in this isolate. + const appPlaneRequest = servedByAppPlane(url.pathname, request.method); + span.setAttribute("executor.dispatch.plane", appPlaneRequest ? "app" : "start"); + if (appPlaneRequest) span.setAttribute("executor.app_graph.entered", appGraphEntered); + // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; observe response/error for span status, keep the flush alive past the response try { - const response = await mcpAgentHandler(new Request(forwarded, { headers }), env, ctx); + const traced = withTraceparent(request, span.spanContext()); + const response = appPlaneRequest + ? await (await getAppPlane()).handler(prepareMcpOrgScope(traced)) + : await fetchHandler(traced, env, ctx); span.setAttribute(ATTR_HTTP_RESPONSE_STATUS_CODE, response.status); - if (response.status >= 500) { - span.setStatus({ code: SpanStatusCode.ERROR }); - } return response; } catch (err) { - span.setStatus({ code: SpanStatusCode.ERROR }); + // oxlint-disable-next-line executor/no-instanceof-error, executor/no-unknown-error-message -- adapter boundary: Cloudflare's fetch callback throws untyped; normalized only for the OTel span record, the original error is rethrown below + const cause = err instanceof Error ? err : String(err); + span.recordException(cause); + // oxlint-disable-next-line executor/no-unknown-error-message -- adapter boundary: same normalization as the recordException line above + const message = typeof cause === "string" ? cause : cause.message; + span.setStatus({ code: SpanStatusCode.ERROR, message }); // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; preserve original error to Cloudflare runtime throw err; } finally { span.end(); + // The flush still must outlive the request — Effect's + // BatchSpanProcessor ships on a timer. ctx.waitUntil(flushTracerProvider()); } }, ); } - if (!tracingInstalled) { - return fetchHandler(request, env, ctx); - } - // Effect-served paths bring their own http.server span (with traceparent - // join) — opening one here too would duplicate it. See the header note. - if (isAppOwnedPath(url.pathname)) { - // The provider is installed (above) and the flush still must outlive - // the request — Effect's BatchSpanProcessor ships on a timer. - // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; mirror the traced path's finally - try { - return await fetchHandler(request, env, ctx); - } finally { - ctx.waitUntil(flushTracerProvider()); - } - } return tracer.startActiveSpan( `http.server ${request.method}`, { kind: SpanKind.SERVER }, @@ -237,11 +412,18 @@ const cloudflareHandler: ExportedHandler = { const response = await fetchHandler(request, env, ctx); span.setAttribute(ATTR_HTTP_RESPONSE_STATUS_CODE, response.status); if (response.status >= 500) { - span.setStatus({ code: SpanStatusCode.ERROR }); + span.setStatus({ code: SpanStatusCode.ERROR, message: `HTTP ${response.status}` }); } return response; } catch (err) { - span.setStatus({ code: SpanStatusCode.ERROR }); + // Record the exception itself, not just the status bit: without it + // these spans are ERROR with zero diagnostic content. + // oxlint-disable-next-line executor/no-instanceof-error, executor/no-unknown-error-message -- adapter boundary: Cloudflare's fetch callback throws untyped; normalized only for the OTel span record, the original error is rethrown below + const cause = err instanceof Error ? err : String(err); + span.recordException(cause); + // oxlint-disable-next-line executor/no-unknown-error-message -- adapter boundary: same normalization as the recordException line above + const message = typeof cause === "string" ? cause : cause.message; + span.setStatus({ code: SpanStatusCode.ERROR, message }); // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; preserve original error to Cloudflare runtime throw err; } finally { @@ -253,4 +435,4 @@ const cloudflareHandler: ExportedHandler = { }, }; -export default Sentry.withSentry(sentryOptions, cloudflareHandler); +export default Sentry.withSentry(cloudSentryOptions, cloudflareHandler); diff --git a/apps/cloud/src/start.ts b/apps/cloud/src/start.ts index a35a327f1..aaadc1a52 100644 --- a/apps/cloud/src/start.ts +++ b/apps/cloud/src/start.ts @@ -1,16 +1,14 @@ import { createMiddleware, createStart } from "@tanstack/react-start"; import { decodeOAuthCallbackState } from "@executor-js/sdk/shared"; -import { cloudApiHandler } from "./app"; import { isAppOwnedPath } from "./app-paths"; -import { authGateMiddleware } from "./auth/ssr-gate"; +import { authGateMiddleware } from "./auth/doc-gate"; import { parseCookie } from "./auth/cookies"; import { ORG_SELECTOR_HEADER } from "./auth/organization"; import { loginPath } from "./auth/return-to"; import { prepareMcpOrgScope } from "./mcp/mount"; import { docsProxyMiddleware, - marketingMiddleware, openAiAppsChallengeMiddleware, posthogProxyMiddleware, sentryTunnelMiddleware, @@ -36,8 +34,18 @@ import { // (a workerd-only virtual module) into the browser build, breaking it. Keeping // the call inside the server callback mirrors how every other server concern // here stays server-only. -let app: ReturnType | undefined; -const getApp = () => (app ??= cloudApiHandler()); +// +// The IMPORT is dynamic for a second, server-side reason: `server.ts` now +// dispatches `/api/*` at the Worker entry, so the only paths that still reach +// this middleware are the two Start's own chain claims first (the Sentry tunnel +// and the OAuth callback's signed-out redirect) plus `/mcp`. A static import +// would still put the entire Effect app in the graph every SSR page load +// evaluates — 3.06 MB of the 12.94 MB page closure, for code a page never runs. +// Deferring it takes the page closure to 9.88 MB; `scripts/start-closure.mjs` +// reports both planes and will show it coming back if this becomes static. +let app: ReturnType | undefined; +const getApp = async (): Promise> => + (app ??= (await import("./app")).cloudApiHandler()); const SESSION_COOKIE = "wos-session"; const OAUTH_CALLBACK_PATH = "/api/oauth/callback"; @@ -79,30 +87,29 @@ const oauthCallbackSignInMiddleware = createMiddleware({ type: "request" }).serv // envelope routes, pinning the org in an internal header (a no-op for everything // else, including `/api/*`). const appRequestMiddleware = createMiddleware({ type: "request" }).server( - ({ pathname, request, next }) => { + async ({ pathname, request, next }) => { if (isAppOwnedPath(pathname)) { const scopedRequest = pathname === OAUTH_CALLBACK_PATH ? oauthCallbackOrgScopedRequest(request) : request; - return getApp().handler(prepareMcpOrgScope(scopedRequest)); + return (await getApp()).handler(prepareMcpOrgScope(scopedRequest)); } return next(); }, ); -// The edge concerns (marketing proxy, docs proxy, sentry tunnel, posthog proxy) -// live in `./edge`; they run before the app's own dispatch. Ordering is -// load-bearing: marketing first (production landing/page proxy), then the docs -// proxy and analytics tunnels, then the unified app plane (api + mcp), and last -// the SSR auth gate — it only sees document requests nothing above claimed, so -// signed-out visitors are redirected to /login before the SPA (and its -// app-shell skeleton) is served. The docs proxy sits among the edges (not after -// the auth gate) because `/docs` is public and must skip the sign-in redirect; -// its path is disjoint from every other matcher, so its slot is not otherwise -// load-bearing. +// The remaining edge concerns (docs proxy, sentry tunnel, posthog proxy) live +// in `./edge`; they run before the app's own dispatch. Marketing is handled in +// server.ts before this module is loaded. Ordering here is load-bearing: public +// challenges and docs, then analytics tunnels, then the unified app plane (api +// + mcp), and last the SSR auth gate — it only sees document requests nothing +// above claimed, so signed-out visitors are redirected to /login before the SPA +// (and its app-shell skeleton) is served. The docs proxy sits among the edges +// (not after the auth gate) because `/docs` is public and must skip the sign-in +// redirect; its path is disjoint from every other matcher, so its slot is not +// otherwise load-bearing. export const startInstance = createStart(() => ({ requestMiddleware: [ openAiAppsChallengeMiddleware, - marketingMiddleware, docsProxyMiddleware, sentryTunnelMiddleware, posthogProxyMiddleware, diff --git a/apps/cloud/src/test-globalsetup-exit.node.test.ts b/apps/cloud/src/test-globalsetup-exit.node.test.ts new file mode 100644 index 000000000..8eff5a4d2 --- /dev/null +++ b/apps/cloud/src/test-globalsetup-exit.node.test.ts @@ -0,0 +1,41 @@ +import { spawnSync } from "node:child_process"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "@effect/vitest"; + +const appRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const vitestBin = resolve(appRoot, "../../node_modules/vitest/vitest.mjs"); +const fixtureConfig = resolve(appRoot, "test-fixtures/test-globalsetup-exit/vitest.config.ts"); + +const runFixture = (port: number, shouldPass: boolean) => + spawnSync(process.execPath, [vitestBin, "run", "--config", fixtureConfig], { + cwd: appRoot, + encoding: "utf8", + timeout: 60_000, + env: { + ...process.env, + CLOUD_TEST_DB_PORT: String(port), + TEST_GLOBALSETUP_SHOULD_PASS: String(shouldPass), + }, + }); + +const diagnostic = (result: ReturnType): string => + [result.stdout, result.stderr].filter(Boolean).join("\n"); + +describe("cloud test global setup", () => { + it("does not let PGlite teardown turn a passed test red", { timeout: 60_000 }, () => { + const result = runFixture(45_435, true); + + expect(result.error).toBeUndefined(); + expect(result.signal).toBeNull(); + expect(result.status, diagnostic(result)).toBe(0); + }); + + it("does not let PGlite teardown turn a failed test green", { timeout: 60_000 }, () => { + const result = runFixture(45_436, false); + + expect(result.error).toBeUndefined(); + expect(result.signal).toBeNull(); + expect(result.status, diagnostic(result)).toBe(1); + }); +}); diff --git a/apps/cloud/test-fixtures/test-globalsetup-exit/fixture.test.ts b/apps/cloud/test-fixtures/test-globalsetup-exit/fixture.test.ts new file mode 100644 index 000000000..453b8f8b8 --- /dev/null +++ b/apps/cloud/test-fixtures/test-globalsetup-exit/fixture.test.ts @@ -0,0 +1,7 @@ +import { expect, it } from "@effect/vitest"; + +it("exercises the global-setup exit path", () => { + if (process.env.TEST_GLOBALSETUP_SHOULD_PASS === "true") return; + + expect("deliberate failure").toBe("reported as success"); +}); diff --git a/apps/cloud/test-fixtures/test-globalsetup-exit/vitest.config.ts b/apps/cloud/test-fixtures/test-globalsetup-exit/vitest.config.ts new file mode 100644 index 000000000..e073da19f --- /dev/null +++ b/apps/cloud/test-fixtures/test-globalsetup-exit/vitest.config.ts @@ -0,0 +1,12 @@ +import { resolve } from "node:path"; +import { defineConfig } from "vitest/config"; + +const appRoot = resolve(__dirname, "../.."); + +export default defineConfig({ + root: appRoot, + test: { + include: ["test-fixtures/test-globalsetup-exit/fixture.test.ts"], + globalSetup: [resolve(appRoot, "scripts/test-globalsetup.ts")], + }, +}); diff --git a/apps/cloud/vite.config.ts b/apps/cloud/vite.config.ts index ec565da2f..308d5d60d 100644 --- a/apps/cloud/vite.config.ts +++ b/apps/cloud/vite.config.ts @@ -142,6 +142,16 @@ export default defineConfig(({ command, mode }) => { router: { virtualRouteConfig: routes, }, + // SPA mode: the console is 100% authenticated UI (marketing is its own + // Astro app, docs are a proxy), so nothing needs per-request React + // SSR. The shell is prerendered once at build; document requests still + // run the request-middleware chain (doc-gate auth redirects + session + // cookie rotation) but serve that static shell, which drops the whole + // React app from the worker bundle and takes per-request render cost + // to zero. + spa: { + enabled: true, + }, }), react(), ], diff --git a/apps/cloud/wrangler.jsonc b/apps/cloud/wrangler.jsonc index dfd9bbd80..3578de6d2 100644 --- a/apps/cloud/wrangler.jsonc +++ b/apps/cloud/wrangler.jsonc @@ -21,6 +21,12 @@ "observability": { "enabled": true, }, + // Script-level logpush feeds the account's workers_trace_events Logpush job + // (invocation logs, outcomes like exceededMemory, console output) into + // Axiom. Pinned here because the setting lives on the script: a deploy that + // omits it can reset it to false, which is how the export silently died in + // the July deploy-tooling migration. + "logpush": true, "durable_objects": { "bindings": [ { @@ -88,6 +94,11 @@ "bucket_name": "executor-cloud-blobs", }, ], + // Pins the Worker to the Hyperdrive/Postgres region. Added in #420 to fix + // MCP Hyperdrive connection pressure; dropped in #1693 as a placement + // experiment, which took `user_store.*` from ~12ms to ~300ms because every + // query started crossing regions. Restored to measure that against the + // JWKS verify time the experiment also moved. "placement": { "region": "aws:us-east-1", }, @@ -96,8 +107,22 @@ "binding": "LOADER", }, ], + // Stamps spans (`service.version`) and mem-metrics snapshots with the + // running Worker version id, so an Axiom step-change maps to the exact + // deploy without Cloudflare API archaeology. memory-metrics.ts already + // reads this binding; it was never declared before, so it always fell back. + "version_metadata": { + "binding": "CF_VERSION_METADATA", + }, "vars": { "VITE_PUBLIC_SITE_URL": "https://executor.sh", + // Keeps the /__sentry-otel-verify probe live in production: Sentry + // delivery failed silently for weeks (zero events after ~Jul 30 with an + // active DSN), and without this there is no way to test the pipeline + // end-to-end short of waiting for a real error. The endpoint only fires + // when its exact path is requested and the events are explicitly tagged + // synthetic. + "SENTRY_OTEL_VERIFY": "true", "VITE_PUBLIC_POSTHOG_KEY": "phc_nNLrNMALpRsfrEkZovUkfMxYbcJvHnsJHeoSPavprgLL", // Browser OTLP spans → same-origin, forwarded to Axiom by the worker // (src/observability/browser-traces.ts). Relative on purpose: the diff --git a/apps/desktop/CHANGELOG.md b/apps/desktop/CHANGELOG.md index fafcf3221..1eabb33d0 100644 --- a/apps/desktop/CHANGELOG.md +++ b/apps/desktop/CHANGELOG.md @@ -1,5 +1,11 @@ # @executor-js/desktop +## 1.6.0 + +## 1.5.42 + +## 1.5.41 + ## 1.5.40 ## 1.5.39 diff --git a/apps/desktop/build/entitlements.mac.plist b/apps/desktop/build/entitlements.mac.plist deleted file mode 100644 index 043ecf496..000000000 --- a/apps/desktop/build/entitlements.mac.plist +++ /dev/null @@ -1,21 +0,0 @@ - - - - - - com.apple.security.cs.allow-jit - - com.apple.security.cs.allow-unsigned-executable-memory - - - com.apple.security.cs.allow-dyld-environment-variables - - - com.apple.security.cs.disable-library-validation - - - diff --git a/apps/desktop/build/icon.png b/apps/desktop/build/icon.png deleted file mode 100644 index 48dadbfc5..000000000 Binary files a/apps/desktop/build/icon.png and /dev/null differ diff --git a/apps/desktop/package.json b/apps/desktop/package.json index 57e96c240..d068c0739 100644 --- a/apps/desktop/package.json +++ b/apps/desktop/package.json @@ -1,6 +1,6 @@ { "name": "@executor-js/desktop", - "version": "1.5.40", + "version": "1.6.0", "private": true, "homepage": "https://github.com/UsefulSoftwareCo/executor", "license": "MIT", diff --git a/apps/desktop/src/main/crash-fingerprint.test.ts b/apps/desktop/src/main/crash-fingerprint.test.ts new file mode 100644 index 000000000..057498f9c --- /dev/null +++ b/apps/desktop/src/main/crash-fingerprint.test.ts @@ -0,0 +1,143 @@ +import { expect, test } from "@effect/vitest"; + +import { + crashReportFingerprint, + mainCrashReportingOptions, + withCrashReportFingerprint, + type CrashEvent, +} from "./crash-fingerprint"; + +/** A crash event as Sentry hands it to `beforeSend` — with the grouping key + * slot the hook is allowed to fill. */ +type SentCrashEvent = CrashEvent & { readonly fingerprint?: readonly string[] | undefined }; + +// Chromium's soft-assert path (`NOTREACHED()`/`DCHECK`) dumps and keeps +// running. Sentry titles each one with the faulting load address, so one +// condition arrives as a new issue every time the address moves. +const softAssertEvent = (address: string): SentCrashEvent => ({ + exception: { + values: [ + { + type: "Fatal Error", + value: `Simulated Exception / ${address}`, + mechanism: { type: "minidump" }, + stacktrace: { + frames: [ + { function: "logging::NotReachedLogMessage::~NotReachedLogMessage" }, + { function: "logging::HandleCheckErrorLogMessage" }, + { function: "base::debug::DumpWithoutCrashing" }, + { function: "crash_reporter::DumpWithoutCrashing" }, + ], + }, + }, + ], + }, +}); + +test("address-keyed Chromium soft asserts collapse to one fingerprint", () => { + const first = crashReportFingerprint(softAssertEvent("0x00000001a2b3c4d5")); + const second = crashReportFingerprint(softAssertEvent("0x00000007e8f9a0b1")); + + expect(first).toEqual(["chromium-dump-without-crashing"]); + expect(first).toEqual(second); +}); + +test("a real native abort keeps Sentry's own grouping", () => { + const abortEvent: CrashEvent = { + exception: { + values: [ + { + type: "EXC_CRASH", + value: "SIGABRT", + mechanism: { type: "minidump" }, + stacktrace: { + frames: [ + { function: "abort" }, + { function: "pthread_kill" }, + { function: "__pthread_kill" }, + ], + }, + }, + ], + }, + }; + + expect(crashReportFingerprint(abortEvent)).toBeUndefined(); +}); + +test("renderer chunk hashes are normalized out of the fingerprint", () => { + const rendererEvent = (chunkHash: string): CrashEvent => ({ + culprit: `loadConnections(assets/atoms-${chunkHash})`, + exception: { + values: [ + { + type: "TypeError", + value: "cannot read properties of undefined", + stacktrace: { + frames: [ + { + filename: `http://127.0.0.1:4789/assets/atoms-${chunkHash}.js`, + module: `atoms-${chunkHash}`, + function: "loadConnections", + in_app: true, + }, + ], + }, + }, + ], + }, + }); + + const release1 = crashReportFingerprint(rendererEvent("Yemn7yhP")); + const release2 = crashReportFingerprint(rendererEvent("CeCENfWa")); + + expect(release1).toEqual(["TypeError", "loadConnections@atoms"]); + expect(release1).toEqual(release2); +}); + +// `withCrashReportFingerprint` is the function object diagnostics.ts installs +// as its `beforeSend`, so these assertions cover the main-process wiring and +// not just the classifier behind it. +test("the main-process beforeSend pins the key and forwards the event", () => { + const collapsed = withCrashReportFingerprint(softAssertEvent("0x00000001a2b3c4d5")); + expect(collapsed.fingerprint).toEqual(["chromium-dump-without-crashing"]); + // Nothing else about the event is touched — this is grouping only. + expect(collapsed.exception?.values?.[0]?.value).toBe("Simulated Exception / 0x00000001a2b3c4d5"); + + const untouched: SentCrashEvent = { + exception: { values: [{ type: "EXC_CRASH", stacktrace: { frames: [{ function: "abort" }] } }] }, + }; + expect(withCrashReportFingerprint(untouched)).toBe(untouched); +}); + +// The wiring check: this is the exact object handed to `Sentry.init` in +// diagnostics.ts. If the hook is ever dropped from the main process, this +// fails. +test("the options the main process installs carry the fingerprinting hook", () => { + const options = mainCrashReportingOptions({ + dsn: "https://public@example.invalid/1", + release: "executor-desktop@0.0.0", + environment: "production", + runId: "abcdef123456", + }); + const sent = options.beforeSend(softAssertEvent("0x00000001a2b3c4d5")); + expect(sent.fingerprint).toEqual(["chromium-dump-without-crashing"]); +}); + +test("events with no volatile grouping input are left alone", () => { + expect(crashReportFingerprint({})).toBeUndefined(); + expect( + crashReportFingerprint({ + exception: { + values: [ + { + type: "Error", + stacktrace: { + frames: [{ filename: "/src/main/sidecar.ts", function: "startSidecar" }], + }, + }, + ], + }, + }), + ).toBeUndefined(); +}); diff --git a/apps/desktop/src/main/crash-fingerprint.ts b/apps/desktop/src/main/crash-fingerprint.ts new file mode 100644 index 000000000..105f60a10 --- /dev/null +++ b/apps/desktop/src/main/crash-fingerprint.ts @@ -0,0 +1,80 @@ +/** + * Grouping keys for desktop crash reports. + * + * Two things split one desktop problem across many Sentry issues: + * + * - Chromium's soft-assert path (`NOTREACHED()`/`DCHECK`) calls + * `DumpWithoutCrashing`, which files a minidump and lets the process carry + * on. Sentry titles each one with the faulting load address, so the same + * condition arrives as a new issue every time the address moves. + * - Renderer and main bundles ship as content-hashed chunks, so unresolved + * frames name `atoms-` and re-group on every release. + * + * Both are grouping problems only: nothing here drops, downgrades or edits an + * event, and the frames keep their hashes so sourcemap resolution still works. + */ + +import { + stableGroupingFingerprint, + withStableGroupingFingerprint, + type GroupingEvent, + type GroupingFrame, +} from "@executor-js/sdk/sentry-grouping"; + +export type CrashEvent = GroupingEvent; + +export const CHROMIUM_SOFT_ASSERT_FINGERPRINT = "chromium-dump-without-crashing"; + +const isSoftAssertFrame = (frame: GroupingFrame): boolean => + (frame.function ?? "").includes("DumpWithoutCrashing"); + +/** + * The fingerprint to attach to a crash event, or `undefined` to keep Sentry's + * default grouping (which is right for a real native abort — one issue per + * distinct stack is what we want there). + */ +export const crashReportFingerprint = (event: CrashEvent): readonly string[] | undefined => { + const frames = (event.exception?.values ?? []).flatMap((value) => value.stacktrace?.frames ?? []); + if (frames.some(isSoftAssertFrame)) return [CHROMIUM_SOFT_ASSERT_FINGERPRINT]; + return stableGroupingFingerprint(event); +}; + +/** + * The `beforeSend` the Electron main process installs. Grouping only — an + * event with no volatile grouping input is forwarded unchanged. + */ +export const withCrashReportFingerprint = (event: T): T => { + const frames = (event.exception?.values ?? []).flatMap((value) => value.stacktrace?.frames ?? []); + if (frames.some(isSoftAssertFrame)) { + return { ...event, fingerprint: [CHROMIUM_SOFT_ASSERT_FINGERPRINT] }; + } + return withStableGroupingFingerprint(event); +}; + +/** + * The Sentry options the Electron main process installs, assembled here rather + * than inline at the `Sentry.init` call so the `beforeSend` wiring is covered + * by crash-fingerprint.test.ts. `diagnostics.ts` only passes this through. + * + * Typed structurally on purpose: this module stays importable without pulling + * in electron, which is what keeps it unit-testable at all. + */ +export const mainCrashReportingOptions = (config: { + readonly dsn: string; + readonly release: string; + readonly environment: string; + readonly runId: string; +}) => ({ + dsn: config.dsn, + release: config.release, + environment: config.environment, + initialScope: { + tags: { + platform: process.platform, + arch: process.arch, + runId: config.runId, + }, + }, + // Grouping only — the event is forwarded untouched otherwise. + beforeSend: withCrashReportFingerprint, +}); diff --git a/apps/desktop/src/main/diagnostics.ts b/apps/desktop/src/main/diagnostics.ts index 3d04b1cc2..c10127a47 100644 --- a/apps/desktop/src/main/diagnostics.ts +++ b/apps/desktop/src/main/diagnostics.ts @@ -23,6 +23,7 @@ import { dirname, join } from "node:path"; import { app, crashReporter, dialog, shell } from "electron"; import log from "electron-log/main.js"; import * as Sentry from "@sentry/electron/main"; +import { mainCrashReportingOptions } from "./crash-fingerprint"; import { getServerSettings } from "./settings"; const sentryDsn = __EXECUTOR_SENTRY_DSN__; @@ -78,18 +79,14 @@ export const sidecarCrashReportingEnv = (): Record => */ export const initErrorReporting = () => { if (errorReportingEnabled) { - Sentry.init({ - dsn: sentryDsn, - release: releaseTag(), - environment: environmentTag(), - initialScope: { - tags: { - platform: process.platform, - arch: process.arch, - runId, - }, - }, - }); + Sentry.init( + mainCrashReportingOptions({ + dsn: sentryDsn, + release: releaseTag(), + environment: environmentTag(), + runId, + }), + ); } else { // No DSN baked in — keep native crash dumps local so a user-reported // crash still leaves minidumps for the diagnostics zip to collect. diff --git a/apps/host-cloudflare/src/mcp/agent-handler.ts b/apps/host-cloudflare/src/mcp/agent-handler.ts index 5ec09fc1b..a87027740 100644 --- a/apps/host-cloudflare/src/mcp/agent-handler.ts +++ b/apps/host-cloudflare/src/mcp/agent-handler.ts @@ -11,6 +11,7 @@ import { currentPropagationHeaders, readArtifactsEnabled, readElicitationMode, + readSearchToolsEnabled, withVerifiedIdentityHeaders, } from "@executor-js/cloudflare/mcp/do-headers"; import type { McpSessionProps } from "@executor-js/cloudflare/mcp/agent-durable-object"; @@ -80,6 +81,7 @@ const propsForPrincipal = ( userId: principal.accountId, elicitationMode: readElicitationMode(request), artifactsEnabled: readArtifactsEnabled(request), + searchToolsEnabled: readSearchToolsEnabled(request), // host-cloudflare only routes the bare `/mcp` endpoint to the Agent // bridge (see worker.ts), so the session always serves the default // resource. diff --git a/apps/host-cloudflare/src/mcp/session-durable-object.ts b/apps/host-cloudflare/src/mcp/session-durable-object.ts index e75199dc0..4fccd6af9 100644 --- a/apps/host-cloudflare/src/mcp/session-durable-object.ts +++ b/apps/host-cloudflare/src/mcp/session-durable-object.ts @@ -102,9 +102,14 @@ export class McpSessionDO extends McpAgentSessionDOBase handle.close() }; } - protected override resolveSessionMeta(token: McpSessionInit): Effect.Effect { + protected override resolveSessionMeta( + token: McpSessionInit, + _storedMeta: SessionMeta | null, + ): Effect.Effect { // Single-tenant: every Access principal belongs to the one configured org, - // so there is nothing to resolve — stamp the configured org name. + // so there is nothing to resolve — stamp the configured org name. Nothing + // to reuse from the stored meta either; config is already the cheapest and + // freshest source there is. return Effect.succeed({ organizationId: token.organizationId, organizationName: this.cfConfig.organizationName, @@ -113,6 +118,7 @@ export class McpSessionDO extends McpAgentSessionDOBase = {}; +export class WorkerEntrypoint {} +export class DurableObject {} +export class WorkflowEntrypoint {} +export class RpcTarget {} +export const exports: Record = {}; diff --git a/apps/host-cloudflare/vitest.config.ts b/apps/host-cloudflare/vitest.config.ts index 5bfa2d586..44cc8cc15 100644 --- a/apps/host-cloudflare/vitest.config.ts +++ b/apps/host-cloudflare/vitest.config.ts @@ -1,6 +1,13 @@ +import { resolve } from "node:path"; + import { defineConfig } from "vitest/config"; export default defineConfig({ + resolve: { + alias: { + "cloudflare:workers": resolve(__dirname, "./test-stubs/cloudflare-workers.ts"), + }, + }, test: { include: ["src/**/*.test.ts"], passWithNoTests: true, diff --git a/apps/host-selfhost/CHANGELOG.md b/apps/host-selfhost/CHANGELOG.md index 808bc643f..09803bcb1 100644 --- a/apps/host-selfhost/CHANGELOG.md +++ b/apps/host-selfhost/CHANGELOG.md @@ -1,5 +1,68 @@ # @executor-js/host-selfhost +## 0.0.42 + +### Patch Changes + +- Updated dependencies [[`c11bef2`](https://github.com/UsefulSoftwareCo/executor/commit/c11bef2cd049db7bbf51b15e18761b14acccb534), [`46cea2c`](https://github.com/UsefulSoftwareCo/executor/commit/46cea2cbb1f414ae58ac876819a51b11967909a6), [`a2d1417`](https://github.com/UsefulSoftwareCo/executor/commit/a2d141758e478274813c8c24d354e1fd0f66af49), [`2bdbedf`](https://github.com/UsefulSoftwareCo/executor/commit/2bdbedf257f54d7c209e8c856c618174c10d6bb3), [`0b0b74f`](https://github.com/UsefulSoftwareCo/executor/commit/0b0b74f673b8098c5248159be36c648097f3c87b), [`256e25e`](https://github.com/UsefulSoftwareCo/executor/commit/256e25e7b291b0c023bc7547d092004b66781bba)]: + - @executor-js/plugin-mcp@1.6.0 + - @executor-js/plugin-openapi@1.6.0 + - @executor-js/sdk@1.6.0 + - @executor-js/react@1.4.63 + - @executor-js/plugin-provider-service-split@0.0.14 + - @executor-js/app@1.4.4 + - @executor-js/analytics@0.1.7 + - @executor-js/api@1.4.63 + - @executor-js/execution@1.6.0 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.11 + - @executor-js/plugin-encrypted-secrets@0.0.42 + - @executor-js/plugin-graphql@1.6.0 + - @executor-js/plugin-toolkits@1.5.35 + - @executor-js/runtime-quickjs@1.6.0 + +## 0.0.41 + +### Patch Changes + +- Updated dependencies [[`d3f0617`](https://github.com/UsefulSoftwareCo/executor/commit/d3f0617deec06c57e0d6e1479fe668f79daf977d), [`32206c7`](https://github.com/UsefulSoftwareCo/executor/commit/32206c7f78654f638bfd27c25c71c30c3d6354be), [`9ecc7cb`](https://github.com/UsefulSoftwareCo/executor/commit/9ecc7cb8b30375ffa960e3fefe4d211e0254e691)]: + - @executor-js/sdk@1.5.42 + - @executor-js/plugin-openapi@1.5.42 + - @executor-js/plugin-mcp@1.5.42 + - @executor-js/app@1.4.4 + - @executor-js/analytics@0.1.6 + - @executor-js/api@1.4.62 + - @executor-js/execution@1.5.42 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.10 + - @executor-js/plugin-encrypted-secrets@0.0.41 + - @executor-js/plugin-graphql@1.5.42 + - @executor-js/plugin-provider-service-split@0.0.13 + - @executor-js/plugin-toolkits@1.5.34 + - @executor-js/react@1.4.62 + - @executor-js/runtime-quickjs@1.5.42 + +## 0.0.40 + +### Patch Changes + +- Updated dependencies [[`d572658`](https://github.com/UsefulSoftwareCo/executor/commit/d572658d74097917412256f10a3ea2e3974f44dd), [`a9b33d2`](https://github.com/UsefulSoftwareCo/executor/commit/a9b33d25c32fbb4a292b7e8963e22392f862a16f)]: + - @executor-js/sdk@1.5.41 + - @executor-js/plugin-mcp@1.5.41 + - @executor-js/app@1.4.4 + - @executor-js/analytics@0.1.5 + - @executor-js/api@1.4.61 + - @executor-js/execution@1.5.41 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.9 + - @executor-js/plugin-encrypted-secrets@0.0.40 + - @executor-js/plugin-graphql@1.5.41 + - @executor-js/plugin-openapi@1.5.41 + - @executor-js/plugin-provider-service-split@0.0.12 + - @executor-js/plugin-toolkits@1.5.33 + - @executor-js/react@1.4.61 + - @executor-js/runtime-quickjs@1.5.41 + ## 0.0.39 ### Patch Changes diff --git a/apps/host-selfhost/package.json b/apps/host-selfhost/package.json index 9a6f70ce1..a78ce49f7 100644 --- a/apps/host-selfhost/package.json +++ b/apps/host-selfhost/package.json @@ -1,6 +1,6 @@ { "name": "@executor-js/host-selfhost", - "version": "0.0.39", + "version": "0.0.42", "private": true, "type": "module", "exports": { diff --git a/apps/host-selfhost/src/config.ts b/apps/host-selfhost/src/config.ts index e0cd282d5..dae13a5e1 100644 --- a/apps/host-selfhost/src/config.ts +++ b/apps/host-selfhost/src/config.ts @@ -43,6 +43,14 @@ export interface SelfHostConfig { readonly organizationName: string; /** URL slug for org-prefixed console paths (`//policies`). */ readonly orgSlug: string; + /** + * Sandbox execution budget passed to the QuickJS runtime, or undefined for + * the runtime's own default (5 minutes). An operator knob in principle, but + * its real consumer is the e2e harness, which shrinks it to seconds so the + * sandbox-deadline scenario proves its race without waiting out real + * minutes (the same pattern as MCP_PAUSED_SESSION_IDLE_TIMEOUT_MS on cloud). + */ + readonly sandboxTimeoutMs: number | undefined; } export const resolveDataDir = (): string => @@ -148,9 +156,26 @@ export const loadConfig = (): SelfHostConfig => { bootstrapAdminName: process.env.EXECUTOR_BOOTSTRAP_ADMIN_NAME ?? "Admin", organizationName: process.env.EXECUTOR_ORG_NAME ?? "Default", orgSlug: resolveOrgSlug(), + sandboxTimeoutMs: resolveSandboxTimeoutMs(), }; }; +// A malformed value is refused rather than silently ignored: an operator who +// sets the knob and typos it should find out at boot, not by watching a +// runaway execution use the 5-minute default. +const resolveSandboxTimeoutMs = (): number | undefined => { + const raw = process.env.EXECUTOR_SANDBOX_TIMEOUT_MS; + if (!raw) return undefined; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed <= 0) { + // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- boundary: refuse to boot on a malformed operator knob + throw new Error( + `EXECUTOR_SANDBOX_TIMEOUT_MS ${JSON.stringify(raw)} is not a positive number of milliseconds`, + ); + } + return Math.floor(parsed); +}; + // The org slug doubles as a URL segment (`//policies`), so an // operator-set value must fit the shared grammar and avoid reserved root // segments (api, mcp, login, …) — a colliding slug would shadow real routes. diff --git a/apps/host-selfhost/src/execution.ts b/apps/host-selfhost/src/execution.ts index 270ffc4f8..aa2ffee53 100644 --- a/apps/host-selfhost/src/execution.ts +++ b/apps/host-selfhost/src/execution.ts @@ -65,7 +65,12 @@ export const SelfHostHostConfig: Layer.Layer = Layer.sync(HostConfig export const SelfHostCodeExecutorProvider: Layer.Layer = Layer.sync( CodeExecutorProvider, - () => makeQuickJsExecutor(), + () => { + const { sandboxTimeoutMs } = loadConfig(); + return makeQuickJsExecutor( + sandboxTimeoutMs === undefined ? {} : { timeoutMs: sandboxTimeoutMs }, + ); + }, ); /** diff --git a/apps/local/CHANGELOG.md b/apps/local/CHANGELOG.md index 816ed53fe..755369a8a 100644 --- a/apps/local/CHANGELOG.md +++ b/apps/local/CHANGELOG.md @@ -1,5 +1,86 @@ # @executor-js/local +## 1.6.0 + +### Patch Changes + +- Updated dependencies [[`c11bef2`](https://github.com/UsefulSoftwareCo/executor/commit/c11bef2cd049db7bbf51b15e18761b14acccb534), [`46cea2c`](https://github.com/UsefulSoftwareCo/executor/commit/46cea2cbb1f414ae58ac876819a51b11967909a6), [`a2d1417`](https://github.com/UsefulSoftwareCo/executor/commit/a2d141758e478274813c8c24d354e1fd0f66af49), [`2bdbedf`](https://github.com/UsefulSoftwareCo/executor/commit/2bdbedf257f54d7c209e8c856c618174c10d6bb3), [`0b0b74f`](https://github.com/UsefulSoftwareCo/executor/commit/0b0b74f673b8098c5248159be36c648097f3c87b), [`256e25e`](https://github.com/UsefulSoftwareCo/executor/commit/256e25e7b291b0c023bc7547d092004b66781bba)]: + - @executor-js/plugin-mcp@1.6.0 + - @executor-js/plugin-openapi@1.6.0 + - @executor-js/sdk@1.6.0 + - @executor-js/react@1.4.63 + - @executor-js/plugin-provider-service-split@0.0.14 + - @executor-js/app@1.4.4 + - @executor-js/analytics@0.1.7 + - @executor-js/api@1.4.63 + - @executor-js/config@1.6.0 + - @executor-js/execution@1.6.0 + - @executor-js/vite-plugin@0.0.60 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.11 + - @executor-js/plugin-desktop-settings@1.6.0 + - @executor-js/plugin-example@1.6.0 + - @executor-js/plugin-file-secrets@1.6.0 + - @executor-js/plugin-graphql@1.6.0 + - @executor-js/plugin-keychain@1.6.0 + - @executor-js/plugin-onepassword@1.6.0 + - @executor-js/plugin-toolkits@1.5.35 + - @executor-js/runtime-quickjs@1.6.0 + +## 1.5.42 + +### Patch Changes + +- Updated dependencies [[`d3f0617`](https://github.com/UsefulSoftwareCo/executor/commit/d3f0617deec06c57e0d6e1479fe668f79daf977d), [`32206c7`](https://github.com/UsefulSoftwareCo/executor/commit/32206c7f78654f638bfd27c25c71c30c3d6354be), [`9ecc7cb`](https://github.com/UsefulSoftwareCo/executor/commit/9ecc7cb8b30375ffa960e3fefe4d211e0254e691)]: + - @executor-js/sdk@1.5.42 + - @executor-js/plugin-openapi@1.5.42 + - @executor-js/plugin-mcp@1.5.42 + - @executor-js/app@1.4.4 + - @executor-js/analytics@0.1.6 + - @executor-js/api@1.4.62 + - @executor-js/config@1.5.42 + - @executor-js/execution@1.5.42 + - @executor-js/vite-plugin@0.0.59 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.10 + - @executor-js/plugin-desktop-settings@1.5.42 + - @executor-js/plugin-example@1.5.42 + - @executor-js/plugin-file-secrets@1.5.42 + - @executor-js/plugin-graphql@1.5.42 + - @executor-js/plugin-keychain@1.5.42 + - @executor-js/plugin-onepassword@1.5.42 + - @executor-js/plugin-provider-service-split@0.0.13 + - @executor-js/plugin-toolkits@1.5.34 + - @executor-js/react@1.4.62 + - @executor-js/runtime-quickjs@1.5.42 + +## 1.5.41 + +### Patch Changes + +- Updated dependencies [[`d572658`](https://github.com/UsefulSoftwareCo/executor/commit/d572658d74097917412256f10a3ea2e3974f44dd), [`a9b33d2`](https://github.com/UsefulSoftwareCo/executor/commit/a9b33d25c32fbb4a292b7e8963e22392f862a16f)]: + - @executor-js/sdk@1.5.41 + - @executor-js/plugin-mcp@1.5.41 + - @executor-js/app@1.4.4 + - @executor-js/analytics@0.1.5 + - @executor-js/api@1.4.61 + - @executor-js/config@1.5.41 + - @executor-js/execution@1.5.41 + - @executor-js/vite-plugin@0.0.58 + - @executor-js/host-mcp@1.4.4 + - @executor-js/mcp-apps-shell@1.4.9 + - @executor-js/plugin-desktop-settings@1.5.41 + - @executor-js/plugin-example@1.5.41 + - @executor-js/plugin-file-secrets@1.5.41 + - @executor-js/plugin-graphql@1.5.41 + - @executor-js/plugin-keychain@1.5.41 + - @executor-js/plugin-onepassword@1.5.41 + - @executor-js/plugin-openapi@1.5.41 + - @executor-js/plugin-provider-service-split@0.0.12 + - @executor-js/plugin-toolkits@1.5.33 + - @executor-js/react@1.4.61 + - @executor-js/runtime-quickjs@1.5.41 + ## 1.5.40 ### Patch Changes diff --git a/apps/local/package.json b/apps/local/package.json index 6c5ae9131..e4ab63c2f 100644 --- a/apps/local/package.json +++ b/apps/local/package.json @@ -1,6 +1,6 @@ { "name": "@executor-js/local", - "version": "1.5.40", + "version": "1.6.0", "private": true, "type": "module", "exports": { @@ -46,7 +46,7 @@ "@executor-js/sdk": "workspace:*", "@executor-js/vite-plugin": "workspace:*", "@libsql/client": "catalog:", - "@modelcontextprotocol/sdk": "^1.12.1", + "@modelcontextprotocol/sdk": "^1.29.0", "@tanstack/react-router": "catalog:", "drizzle-orm": "catalog:", "effect": "catalog:", diff --git a/apps/local/src/executor.ts b/apps/local/src/executor.ts index dd91838f3..1ec7ebbf6 100644 --- a/apps/local/src/executor.ts +++ b/apps/local/src/executor.ts @@ -19,7 +19,7 @@ import type { McpPluginExtension } from "@executor-js/plugin-mcp"; import executorConfig from "../executor.config"; import { localAnalytics } from "./analytics"; import { localDataMigrations } from "./db/data-migrations"; -import { openOwnedLocalDatabase } from "./db/owned-database"; +import { openOwnedLocalDatabase, type OwnedLocalDatabase } from "./db/owned-database"; interface ResolvedStorage { readonly dataDir: string; @@ -56,6 +56,16 @@ type LocalPlugins = readonly AnyPlugin[]; export interface LocalExecutorOptions { readonly activeToolkitSlug?: string; + /** + * Reuse an already-open owned database instead of opening (and locking) the + * data dir again. A toolkit-scoped MCP session differs from the default one + * only in its plugin set, so it must ride the running server's DB handle: + * `openOwnedLocalDatabase` takes an EXCLUSIVE lock, and a second open from + * inside the same process contends with the lock this process already holds. + * The borrowed handle is NOT closed when the derived executor disposes — + * whoever opened it still owns its lifetime. + */ + readonly borrowedDb?: OwnedLocalDatabase; } const loadLocalPlugins = (options: LocalExecutorOptions = {}) => @@ -92,6 +102,10 @@ const loadLocalPlugins = (options: LocalExecutorOptions = {}) => interface LocalExecutorBundle { readonly executor: Executor; readonly plugins: LocalPlugins; + /** The owned DB this bundle opened (or borrowed). Surfaced so a + * toolkit-scoped executor can ride the SAME handle instead of contending + * with this process's own exclusive data-dir lock. */ + readonly db: OwnedLocalDatabase; /** Where this daemon's web UI is reachable, resolved once at boot. Surfaced * so callers building user-facing links (MCP artifact deep links) use the * same origin the executor itself was configured with. */ @@ -151,23 +165,27 @@ const createLocalExecutorLayer = (options: LocalExecutorOptions = {}) => { const tenantId = makeTenantId(cwd); const tables = collectTables(); - const owned = yield* Effect.acquireRelease( - Effect.tryPromise({ - try: () => - openOwnedLocalDatabase({ - dataDir: storage.dataDir, - tables, - namespace: localNamespace, - tenantId, + // A borrowed handle is owned by its opener, so it is used as-is and left + // open on release; only a handle opened here is closed here. + const owned = options.borrowedDb + ? options.borrowedDb + : yield* Effect.acquireRelease( + Effect.tryPromise({ + try: () => + openOwnedLocalDatabase({ + dataDir: storage.dataDir, + tables, + namespace: localNamespace, + tenantId, + }), + catch: (cause) => + new LocalExecutorCreateError({ + message: CREATE_SQLITE_ERROR_MESSAGE, + cause, + }), }), - catch: (cause) => - new LocalExecutorCreateError({ - message: CREATE_SQLITE_ERROR_MESSAGE, - cause, - }), - }), - (database) => Effect.promise(() => database.close()).pipe(Effect.ignore), - ); + (database) => Effect.promise(() => database.close()).pipe(Effect.ignore), + ); const sqlite = owned.db; const migration = owned.migration; @@ -243,7 +261,7 @@ const createLocalExecutorLayer = (options: LocalExecutorOptions = {}) => { ); } - return { executor, plugins, webBaseUrl }; + return { executor, plugins, webBaseUrl, db: owned }; }), ); }; @@ -257,6 +275,7 @@ export const createExecutorHandle = async (options: LocalExecutorOptions = {}) = executor: bundle.executor, plugins: bundle.plugins, webBaseUrl: bundle.webBaseUrl, + db: bundle.db, dispose: async () => { await Effect.runPromise(Effect.ignore(bundle.executor.close())); await ignorePromiseFailure("disposeRuntime", () => runtime.dispose()); diff --git a/apps/local/src/main.ts b/apps/local/src/main.ts index 26378f58f..2ef674c57 100644 --- a/apps/local/src/main.ts +++ b/apps/local/src/main.ts @@ -136,8 +136,13 @@ export const createServerHandlers = async (token: string): Promise Promise; } +const viteChildSignals = ["SIGINT", "SIGTERM", "SIGHUP"] as const; + async function allocatePort(): Promise { const probe = Bun.serve({ port: 0, @@ -127,15 +129,15 @@ async function allocatePort(): Promise { async function startViteChild(): Promise { const vitePort = await allocatePort(); const cwd = resolve(import.meta.dirname, ".."); + const viteEntrypoint = resolve(cwd, "node_modules/vite/bin/vite.js"); const env = { ...process.env }; delete env.PORT; - // `bunx --bun vite` runs vite under Bun, matching the `dev:vite` script - // already in apps/local. --strictPort keeps the URL we hand back stable. + // Run Vite directly under Bun, matching the `dev:vite` script without a + // bunx wrapper that can outlive its child. --strictPort keeps the URL stable. const child: Subprocess = Bun.spawn( [ - "bunx", - "--bun", - "vite", + process.execPath, + viteEntrypoint, "dev", "--port", String(vitePort), @@ -158,20 +160,45 @@ async function startViteChild(): Promise { }, ); + let stopping = false; + const stop = async (): Promise => { + if (stopping) { + await child.exited; + return; + } + stopping = true; + for (const signal of viteChildSignals) process.off(signal, stopOnParentSignal); + if (child.exitCode === null) child.kill(); + await Promise.race([child.exited, Bun.sleep(5_000)]); + if (child.exitCode === null) child.kill("SIGKILL"); + await child.exited; + }; + const stopOnParentSignal = (): void => { + // A PTY/session teardown can signal the CLI while Vite is still optimizing + // dependencies, before the server's normal stop handle exists. Reap the + // owned child immediately; the CLI's signal waiter performs full cleanup + // once startup has completed. + void stop(); + }; + for (const signal of viteChildSignals) process.once(signal, stopOnParentSignal); + const url = `http://127.0.0.1:${vitePort}`; const deadline = Date.now() + 30_000; while (Date.now() < deadline) { // oxlint-disable-next-line executor/no-try-catch-or-throw -- boundary: probing a child process that may not be listening yet try { - const r = await fetch(`${url}/`, { redirect: "manual" }); + const r = await fetch(`${url}/`, { + redirect: "manual", + // A listening socket is not proof that Vite can answer. Bound each + // probe so one accepted-but-stalled request cannot defeat the 30s boot + // deadline and wedge the entire local e2e suite. + signal: AbortSignal.timeout(5_000), + }); if (r.status < 500) { await r.body?.cancel(); return { url, - stop: async () => { - child.kill(); - await child.exited; - }, + stop, }; } await r.body?.cancel(); @@ -179,12 +206,13 @@ async function startViteChild(): Promise { // not up yet } if (child.exitCode !== null) { + await stop(); // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- boundary: child process aborted before becoming ready throw new Error(`vite dev exited with code ${child.exitCode} before becoming ready`); } await Bun.sleep(150); } - child.kill(); + await stop(); // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- boundary: vite never became reachable throw new Error(`vite dev did not become reachable on ${url} within 30s`); } diff --git a/apps/marketing/src/assets/yc-backed-by.svg b/apps/marketing/src/assets/yc-backed-by.svg index f3b4d3e3c..6606921ff 100755 --- a/apps/marketing/src/assets/yc-backed-by.svg +++ b/apps/marketing/src/assets/yc-backed-by.svg @@ -1,5 +1,4 @@ - - + @@ -14,17 +13,4 @@ - - - - - - - - - - - - - diff --git a/apps/marketing/src/layouts/Layout.astro b/apps/marketing/src/layouts/Layout.astro index 603613436..d791ecda8 100644 --- a/apps/marketing/src/layouts/Layout.astro +++ b/apps/marketing/src/layouts/Layout.astro @@ -50,9 +50,16 @@ const canonical = new URL(Astro.url.pathname, Astro.site ?? Astro.url).toString( // deploy env, so it's a no-op in dev / unconfigured builds (and the SDK // isn't even shipped, thanks to the dynamic import). Events are proxied // first-party through src/middleware.ts to survive adblockers. - // autocapture and pageviews are off on purpose: we only send the explicit - // events the page fires (e.g. the "Set up with your agent" copy). Flip - // capture_pageview on if you want a denominator for conversion. + // autocapture stays off on purpose: beyond $pageview we only send the + // explicit events the page fires (e.g. the "Set up with your agent" copy). + // + // capture_pageview is ON because this is the campaign landing surface. + // With it off, a visit to executor.sh/?utm_source=... produced no + // $pageview at all, so every UTM-tagged campaign read as zero traffic in + // PostHog — the params only ever reached $web_vitals, which no funnel or + // web-analytics view is built on. Astro serves a full page load per + // navigation, so plain `true` (capture once on init) is right here; + // `history_change` is for SPAs like apps/cloud. const phKey = import.meta.env.PUBLIC_POSTHOG_KEY; if (phKey) { // api_host rides the `/_astro` prefix that the cloud edge forwards to @@ -62,7 +69,7 @@ const canonical = new URL(Astro.url.pathname, Astro.site ?? Astro.url).toString( api_host: `${window.location.origin}/_astro/_ph`, ui_host: import.meta.env.PUBLIC_POSTHOG_HOST ?? "https://us.posthog.com", autocapture: false, - capture_pageview: false, + capture_pageview: true, persistence: "localStorage", }); return posthog; diff --git a/apps/marketing/src/pages/about-executor.astro b/apps/marketing/src/pages/about-executor.astro new file mode 100644 index 000000000..bcc4d5228 --- /dev/null +++ b/apps/marketing/src/pages/about-executor.astro @@ -0,0 +1,158 @@ +--- +const pageTitle = "Executor"; +const pageDescription = + "Executor is a web application that lets people connect AI assistants to Google Workspace, related Google services, and other software, then control the actions those assistants can take."; +--- + + + + + + + + + + + + + + + + Executor + + + +
+

Executor

+

+ Executor is a web application that lets people connect AI assistants to their software + and data. It gives those assistants specific tools to complete tasks the person requests, + while letting the person approve or block actions. +

+
+ +
+
+

What Executor does

+

+ A person can use Executor to connect an AI assistant to Google Workspace and related + Google services. The person can then ask the assistant to manage schedules, email, + files, documents, contacts, tasks, meetings, photos, or website data through Executor. +

+

+ Executor performs only the actions the person requests and permits. It does not give an + AI assistant an unrestricted session in the person's account. +

+
+ +
+

How Executor uses Google data

+

+ When a person chooses to connect a Google service, Executor requests the permissions + needed to provide that connection and uses the resulting data to carry out that + person's instructions. +

+
    +
  • Google Calendar: read, create, update, or remove calendars and events, and manage calendar sharing rules.
    • +
  • Gmail: read and search messages, compose and send mail, manage labels, archive or trash messages, or permanently delete messages only when explicitly requested.
    • +
  • Google Sheets: read spreadsheet data, update cells, ranges, and worksheets, and write Drive-file smart chips.
    • +
  • Google Drive, Docs, Slides, and Forms: find and manage files and folders, edit documents and presentations, and create or read forms and responses.
    • +
  • Google Contacts and Tasks: read or update contacts and contact groups, read available profile fields, other contacts, or a Workspace directory, and manage task lists and tasks.
    • +
  • Google Meet: create and configure meeting spaces or read meeting records, participants, recordings, and transcripts.
    • +
  • Google Photos: upload and manage app-created media or read media explicitly selected through Google Photos Picker.
    • +
  • Google Search Console: inspect verified sites, sitemaps, indexed URLs, and search-performance data.
    • +
+
+ +
+

Privacy and user control

+

+ Executor does not sell Google user data. A person can remove a connection in Executor or + revoke it from their Google Account permissions. More information is available in the + Executor Privacy Policy. +

+

+ Executor's use and transfer of information received from Google APIs adheres to the + Google API Services User Data Policy, including the Limited Use requirements. +

+
+
+ + + + diff --git a/apps/marketing/src/pages/google-oauth.astro b/apps/marketing/src/pages/google-oauth.astro new file mode 100644 index 000000000..a4b080248 --- /dev/null +++ b/apps/marketing/src/pages/google-oauth.astro @@ -0,0 +1,554 @@ +--- +import Layout from "../layouts/Layout.astro"; + +const googleServices = [ + { + index: "01", + name: "Google Calendar", + purpose: + "Executor can read, create, update, or remove calendars and events, and manage calendar sharing rules, when you ask an agent to manage your schedule.", + scope: "googleapis.com/auth/calendar", + }, + { + index: "02", + name: "Gmail", + purpose: + "Executor can read, search, compose, send, organize, trash, and permanently delete messages, and manage filters and other basic Gmail settings, only when you explicitly instruct an agent to work with your email.", + scope: "mail.google.com · gmail.settings.basic", + }, + { + index: "03", + name: "Google Sheets", + purpose: + "Executor can read spreadsheet data, update cells, ranges, and worksheets, and write Drive-file smart chips when you ask an agent to work with a spreadsheet.", + scope: "googleapis.com/auth/spreadsheets · drive.file", + }, + { + index: "04", + name: "Google Drive", + purpose: + "Executor can find, create, download, organize, share, or delete files and folders when you ask an agent to manage your Drive.", + scope: "googleapis.com/auth/drive", + }, + { + index: "05", + name: "Google Docs, Slides, and Forms", + purpose: + "Executor can read and edit documents and presentations, and create or read forms and responses, when you ask an agent to work with that content.", + scope: "documents · presentations · forms.body · forms.responses.readonly", + }, + { + index: "06", + name: "Google Contacts and Tasks", + purpose: + "Executor can read or update contacts and contact groups, read available profile fields, other contacts, or a Workspace directory, and manage task lists and tasks when you ask an agent to organize people or work.", + scope: "contacts · user profile fields · directory.readonly · tasks", + }, + { + index: "07", + name: "Google Meet", + purpose: + "Executor can create and configure meeting spaces or read meeting records, participants, recordings, and transcripts when you request meeting-related work.", + scope: "meetings.space.created · readonly · settings", + }, + { + index: "08", + name: "Google Photos", + purpose: + "Executor can upload and manage app-created media or read media that you explicitly select through Google Photos Picker.", + scope: "photoslibrary.app-created read/write · photospicker.readonly", + }, + { + index: "09", + name: "Google Search Console", + purpose: + "Executor can inspect verified sites, sitemaps, indexed URLs, and search-performance data when you request website analysis.", + scope: "googleapis.com/auth/webmasters", + }, +] as const; +--- + + +
+ + + + +
+
+ + Google Workspace connections +
+

Executor

+

+ Executor is an integration platform and MCP gateway for AI agents. It lets you + securely connect Google Workspace and related Google services so an agent can read + data and take the actions you request. +

+
+ One connection + Explicit permission + User-directed actions +
+
+ +
+
+

01 / Google data

+
+

What Executor accesses—and why

+

+ You choose which Google service to connect. Executor requests only the + permissions needed for that connection and uses them to carry out your + instructions. +

+
+
+ +
+ { + googleServices.map((service) => ( +
+
{service.index}
+
+

{service.name}

+

{service.purpose}

+
+ {service.scope} +
+ )) + } +
+
+ +
+
+

02 / Your control

+
+

Your Google account stays yours

+

+ Connecting Google does not give Executor permission to act on its own. An + agent can use a Google tool only when you direct it to, subject to the + policies and approvals you configure in Executor. +

+
+
+ +
+
+ +

See the action

+

Executor exposes concrete Google actions with their inputs, not an open-ended account session.

+
+
+ +

Set the policy

+

Allow an action, require approval, or block it before an agent can run it.

+
+ +
+
+ +
+

03 / Privacy

+
+
+

Clear limits on Google data

+

+ Executor does not sell Google user data. We use Google data only to provide + and improve the user-facing integration features you request, in accordance + with our Privacy Policy. Executor's use and transfer of information received + from Google APIs adheres to the + Google API Services User Data Policy, including the Limited Use requirements. +

+
+
+ Read the Privacy Policy + Read the Terms of Service +
+
+
+ +
+
+ Executor · Useful Software Company + Google access, explained plainly. +
+
+
+ + + diff --git a/apps/marketing/src/pages/google-workspace.astro b/apps/marketing/src/pages/google-workspace.astro new file mode 100644 index 000000000..75133113a --- /dev/null +++ b/apps/marketing/src/pages/google-workspace.astro @@ -0,0 +1,218 @@ +--- +const services = [ + { + name: "Google Calendar", + description: + "Read, create, update, or remove calendars and events, and manage calendar sharing rules, when you ask an agent to manage your schedule.", + }, + { + name: "Gmail", + description: + "Read, search, compose, send, label, archive, trash, or permanently delete messages, and manage filters and other basic Gmail settings, when you explicitly ask an agent to work with your email.", + }, + { + name: "Google Sheets", + description: + "Read spreadsheet data, update cells, ranges, or worksheets, and write Drive-file smart chips when you ask an agent to work with a spreadsheet.", + }, + { + name: "Google Drive, Docs, Slides, and Forms", + description: + "Find and manage files and folders, edit documents and presentations, and create or read forms and responses when you ask an agent to work with them.", + }, + { + name: "Google Contacts and Tasks", + description: + "Read or update contacts and contact groups, read available profile fields, other contacts, or a Workspace directory, and manage task lists and tasks when you ask an agent to organize people or work.", + }, + { + name: "Google Meet", + description: + "Create and configure meeting spaces or read meeting records, participants, recordings, and transcripts when you request meeting-related work.", + }, + { + name: "Google Photos", + description: + "Upload and manage app-created photos or read media you explicitly select through Google Photos Picker.", + }, + { + name: "Google Search Console", + description: + "Read and manage verified sites, sitemaps, URL inspection results, and search-performance data when you request website analysis.", + }, +] as const; +--- + + + + + + + + + + Executor + + + +
+

Useful Software Company

+

Executor

+

+ Executor is an integration platform and MCP gateway for AI agents. Executor lets you + securely connect Google Workspace and related Google services so an agent can read your + Google data and take only the actions you request. +

+
+ +
+
+

Why Executor requests access to Google data

+

+ You choose which Google service to connect. Executor requests the permissions needed + to provide that connection and uses the resulting Google data to carry out your + instructions. Connecting Google does not give Executor permission to act on its own. +

+
    + { + services.map((service) => ( +
  • +

    {service.name}

    +

    {service.description}

    +
    • + )) + } +
+
+ +
+

You control the connection

+

+ Executor exposes specific Google actions and their inputs. You can allow an action, + require approval, or block it through the policies you configure in Executor. You can + remove a connection in Executor or revoke it from your + Google Account permissions. +

+
+ +
+

Privacy and Google API data

+

+ Executor does not sell Google user data. Executor uses Google data only to provide and + improve the user-facing integration features you request, as described in the + Executor Privacy Policy. +

+

+ Executor's use and transfer of information received from Google APIs adheres to the + Google API Services User Data Policy, including the Limited Use requirements. +

+
+
+ + + + diff --git a/apps/marketing/src/pages/index.astro b/apps/marketing/src/pages/index.astro index cd0940b87..b57258bc7 100644 --- a/apps/marketing/src/pages/index.astro +++ b/apps/marketing/src/pages/index.astro @@ -203,11 +203,14 @@ Source (and the place to start if something breaks): https://github.com/UsefulSo class="rise-4 mt-10 inline-flex" aria-label="Backed by Y Combinator" > + {/* Shadow lives here, not in the SVG: WebKit rasterizes SVG + filter regions in at 1x, blurring the badge on mobile. */} Backed by Y Combinator @@ -615,7 +618,7 @@ Source (and the place to start if something breaks): https://github.com/UsefulSo