Skip to content

feat(observability,cli): name the calls behind event-loop stalls, and stream dev events from frontmcp dev - #1013

Merged
alexmercerpo merged 95 commits into
mainfrom
feat/m10-stall-dev-events
Nov 3, 2026
Merged

alexmercerpo merged 95 commits into
mainfrom
feat/m10-stall-dev-events

Conversation

@frontegg-david

@frontegg-david frontegg-david commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1009

What and why

Proposal items 4.3 (stall attribution) and 5.4 (the frontmcp dev event stream).

The reported problem: when one tool does synchronous work (a large JSON.parse, a regex on a big string, a sync crypto
call), the event loop stalls and every other call, health check and stream on the server waits. The process-stats
collector already samples the loop delay for /metrics, but it cannot say what was running during a stall, and it
cannot tell a GC pause from a blocking tool. During development there was also no record of how long each call took or
why it failed, other than reading the server's logs.

What this PR adds:

  • Stall detection in @frontmcp/observability, opt-in with observability: { stalls: true | { thresholdMs } } (or
    ObservabilityPlugin.init({ stalls })), 200 ms by default, 20 to 60000.
    • A sampler checks the event loop every 10 ms on an unref'd timer, started when the server serves (onReady) and
      stopped at shutdown (onShutdown). A gap of thresholdMs or more between two checks is a stall, measured to within
      10 ms. Node.js only: where perf_hooks is missing the detector does nothing.
    • The process has one detector per threshold: the servers of a splitByApp server, or of standalone apps, share it,
      so each stall is one warning naming the calls of every server, and is counted once.
    • The calls in flight are kept by Around hooks on the execute stage of tools:call-tool,
      resources:read-resource and prompts:get-prompt, while the detector samples: a call stays in flight until its
      execute stage settles,
      whatever its outcome (return, throw, abort), so a failed call is never left "running". A call that timed out stays
      in flight until the execution it left behind (state.abandonedExecution) settles. Each call records its method,
      the entry's full name, its start time and the first 12 hex characters of the session's tracing hash.
    • GC pauses are observed with a PerformanceObserver on gc entries. A stall is reported on the sample after the
      one that measured it, once its GC entries have been delivered. When the stall less its GC time is under the
      threshold, the stall is GC's and the calls in flight are not blamed.
    • Each stall is one warning line, from the CallMonitor logger, naming at most five calls in flight (method, entry,
      running time, session hash; never an input, an answer or a raw session id), and one increment of
      frontmcp_event_loop_stall_seconds_total{cause, entry}: cause is the call's method, gc or unknown, entry is
      a registered entry's name, so the label values are bounded by the server. The counter is served under the
      process category of /metrics.
    • CallMonitor injects the server logger with the new FRONTMCP_LOGGER token of @frontmcp/sdk
      (Symbol.for('frontmcp:LOGGER'), the same logger as FrontMcpLogger). The SDK requires
      @frontmcp/observability, so a server run as ESM gets the CommonJS copy of the SDK inside the plugin (1.9.3: observability records every failure as SERVER_ERROR in ES-module projects (a second @frontmcp/sdk copy) #802), where
      the FrontMcpLogger class is another token.
  • The frontmcp dev event stream.
    • frontmcp dev empties .frontmcp/dev-events.jsonl in the project root at start, and the server appends one JSON
      line per event: { type: 'call', method, entry, durationMs, outcome, error?: { type, code }, session? } for each
      tools/call, resources/read and prompts/get (reported by Finally hooks on finalize, so failures are
      included), and { type: 'stall', stallMs, gcMs, cause, calls } for each stall. --events-file <path> names
      another file; --no-events turns it off.
    • Under frontmcp dev --stdio --serve the server sends the events over its IPC channel
      ({ __frontmcp: 'dev-events', events }) and the bridge, its parent, appends them to the file.
    • Sinks are pluggable: observability.devEvents takes a file, extra sinks ({ name?, write(events), close? })
      and maxQueue. Events go through a bounded queue (1000) drained off the call path: emit never waits, events past
      the bound are dropped with one warning, and a failing sink is logged once and keeps being written to.
    • Only in development: under frontmcp dev (it sets FRONTMCP_DEV_EVENTS_FILE or FRONTMCP_DEV_EVENTS_IPC on the
      server), or with NODE_ENV=development and a file or sink. A server with no observability config loads
      @frontmcp/observability under frontmcp dev for the events only (tracing and logging off); without the package
      it logs one info line saying how to get them. A server, or one of its apps, that installs ObservabilityPlugin
      itself is left as it is: that plugin writes the events.
  • Validation. stalls and devEvents are strict Zod schemas. An invalid value throws
    ObservabilityConfigurationError (OBSERVABILITY_CONFIGURATION, an InternalMcpError) naming each problem, from
    ObservabilityPlugin.init(); @FrontMcp({ observability }) now rethrows it, so the server does not start. Any
    other failure while setting the plugin up (the package fails to load, tracing or logging cannot be wired) still only
    logs a warning, as before.
  • @frontmcp/utils gets appendFile(), used by the events file sink and the bridge.

Different from the plan, and why:

  • The stall metric is frontmcp_event_loop_stall_seconds_total, not frontmcp_event_loop_stall_seconds: it is a
    counter, and every counter the repo serves ends in _total. It carries cause next to entry, so a tool named
    gc and GC time cannot be confused, and tools and resources with the same name stay apart.
  • The hooks live on a provider, CallMonitor, not on ObservabilityPlugin. The plugin provides it only when
    stalls or dev events are on, and the SDK collects hooks from plugin providers, so a server without either gets no
    extra hook; each server's hooks run on its own monitor; and the detector and the stream share one object, avoiding
    a provider-to-provider dependency between dynamic providers that are built in parallel.
  • Around tracks, Finally reports. The in-flight registry needs only the Around hook on execute: its
    next() rejects when the stage fails, so the hook's own finally ends the call. The Finally hooks on finalize
    report each call to the dev event stream, including calls that failed before execute.
  • The bridge writes the file in --serve mode, rather than the server writing it and also sending IPC: one writer
    per mode, and the file survives child restarts as one stream.
  • The detector times its own samples instead of reading monitorEventLoopDelay(). Resetting that histogram between
    the end of a stall and its next tick drops the stall: a sampler that ran first after a stall lost it, which happened
    for about one stall in eight in a loop of blocking tool calls. The gap between two 10 ms samples measures the same
    delay, at the same 10 ms resolution, with no reset and with the stall's exact window.

User-facing change

  • What changed:
    • @FrontMcp({ observability: { stalls, devEvents } }) and ObservabilityPlugin.init({ stalls, devEvents }).
    • New exports of @frontmcp/observability: CallMonitor, StallDetector, InFlightCalls, describeStall,
      countStall, STALL_SECONDS_COUNTER, DevEventStream, JsonlFileSink, IpcDevEventSink,
      ObservabilityConfigurationError, and the types StallDetectionOptions, DevEventsOptions, StallReport,
      StalledCall, CallIdentity, CallMethod, CallRecord, GcPause, StallDetectorOptions, DevEvent,
      DevCallEvent, DevStallEvent, DevEventSink.
    • New export of @frontmcp/sdk: FRONTMCP_LOGGER (Symbol.for('frontmcp:LOGGER')), the server logger under a token
      that a plugin package loaded by another copy of the SDK resolves too.
    • New metric frontmcp_event_loop_stall_seconds_total{cause, entry}, in the process category of /metrics.
    • New warning line Event loop stalled for <n> ms ... (logger CallMonitor).
    • New error code OBSERVABILITY_CONFIGURATION (ObservabilityConfigurationError), thrown for an invalid
      observability.stalls or observability.devEvents value.
    • frontmcp dev and frontmcp dev --stdio: new flags --events-file <path> and --no-events, a new
      [dev] events: .frontmcp/dev-events.jsonl line, and the .frontmcp/dev-events.jsonl file.
    • Under frontmcp dev, a server with no observability config and no ObservabilityPlugin of its own (on the
      server or an app) loads @frontmcp/observability when it is installed, tracing and logging off, and logs one info
      line when it is not.
    • @frontmcp/utils: appendFile(path, content).
  • Before → after:
    • Before: a stall showed only as a higher frontmcp_nodejs_eventloop_lag_seconds, with nothing naming the call
      behind it. After, with stalls on: a warning naming the calls in flight, stall seconds per entry, GC counted apart.
    • Before: frontmcp dev kept no record of calls. After: call timings, failures and stalls in
      .frontmcp/dev-events.jsonl.
    • Before: a server without an observability config never loaded @frontmcp/observability. After, under
      frontmcp dev only, with the package installed: it is loaded with tracing, logging and request logs off, so the
      server gets the plugin's hooks (no-ops with tracing off) and the dev-event hooks, no spans, no log transport, no
      request logs and nothing new on /metrics; this.telemetry exists in tools and throws
      ContextExtensionNotAvailableError when read, as it does for any server with observability.tracing: false, where
      it was undefined. Outside frontmcp dev nothing changes. observability: false or --no-events turns the
      loading off.
    • Before: ObservabilityPlugin.init() (and @FrontMcp({ observability })) had no option it refused; a failure while
      the SDK set the plugin up from observability was logged as a warning and the server started without it. After:
      an invalid stalls or devEvents value throws ObservabilityConfigurationError and the server does not start;
      every other failure while setting the plugin up still only warns.
  • frontmcp.dev pages affected (if known): https://frontmcp.dev/reference/server/observability (new stall detection
    section, the metric), https://frontmcp.dev/reference/cli#frontmcp-dev (dev events, --events-file, --no-events),
    the /metrics page (new counter).
  • Skills catalog entries updated:
    • frontmcp-observability: new reference references/stall-detection.md and example
      examples/stall-detection/find-a-blocking-tool.md; SKILL.md (routing tables, examples, description and
      triggers); references/metrics-endpoint.md (the stall counter); skills-manifest.json.
    • frontmcp-setup: references/setup-project.md, new "Dev events" section under frontmcp dev.
    • frontmcp-development: references/create-plugin.md, injecting the server logger with FRONTMCP_LOGGER.

Tests

  • Unit (red first):
    • @frontmcp/observability: in-flight-calls.spec.ts (success, throw, abort, a timed-out execution that settles
      later or rejects, overlap and pruning), stall-detector.spec.ts (fake clock and fake GC feed: under the
      threshold, a call running through a stall, a call that blocked and ended just before the sample, a call that ended
      before the stall and one that started after it, GC taking up a stall, a partial GC overlap, the 10 ms interval and
      stop, a runtime without perf_hooks, a throwing listener, the real perf_hooks defaults; and, on the real event
      loop, a stall the sampler is the first to see after it ends),
      stall-report.spec.ts (the log line, the five-call bound, the metric labels), dev-events.spec.ts (batching, a
      slow sink and the queue bound, a failing sink logged once, shutdown, the JSONL file sink, the IPC sink),
      observability-options.spec.ts (defaults, dev-only resolution, every validation message, init() failing),
      call-monitor.spec.ts (the provider and its FRONTMCP_LOGGER injection, the declared hooks, tracking per flow,
      one detector shared by the servers of a process, call events with no input or answer, a stall as a warning, a
      metric and a dev event, start and stop), dev-events-server.spec.ts (a real server under
      FRONTMCP_DEV_EVENTS_FILE: call events with no observability config; an ObservabilityPlugin the server or an
      app installs is kept, its request log still written, each call event written once).
    • SDK: dev-events-observability.spec.ts (the config frontmcp dev implies, none when the server or an app
      installs ObservabilityPlugin, the not-installed message, the schema keeps stalls and devEvents),
      logger-token.spec.ts (FRONTMCP_LOGGER is the server logger, under Symbol.for('frontmcp:LOGGER')),
      observability-scope.helper.spec.ts (a configuration error fails the boot, another failure still warns),
      metrics.service.spec.ts (the stall counter under process).
    • CLI: resolve-dev-launch.spec.ts (default file, --events-file from a subfolder, IPC for --serve,
      --no-events clearing an inherited value), dev-events-file.spec.ts, upstream-client.spec.ts (dev events
      from the IPC channel), bridge.spec.ts (flag mapping).
    • @frontmcp/utils: appendFile.
  • E2E:
    • demo-e2e-standalone/e2e/stall-attribution.e2e.spec.ts: a busy-waiting tool is named in the stall warning (with
      its session hash), and a call that failed before it is not. Checked against a build whose registry ends a call
      only on success: the test fails, naming the failed call.
    • demo-e2e-cli-exec/e2e/cli-dev-events.e2e.spec.ts: frontmcp dev writes call timings, a failure and a stall to
      .frontmcp/dev-events.jsonl; frontmcp dev --stdio --serve gets the same events over IPC into
      --events-file; a server that installs ObservabilityPlugin itself keeps it under frontmcp dev (its request
      log is written) and each call event is written once. Checked against a build that adds the events-only plugin
      anyway: the test fails, with no request log.
    • demo-e2e-dts-consumer: stalls-dev-events-must-compile.ts types observability.stalls and devEvents, and a
      provider injecting FRONTMCP_LOGGER.
  • Review: a call tracked while the detector does not sample (no perf_hooks, before onReady, after shutdown) was
    never pruned; the monitor now tracks calls only while the detector samples (StallDetector.running).
  • Second review:
    • Resetting the monitorEventLoopDelay() histogram before its next tick dropped the stall that just ended (about one
      in eight in a loop of blocking calls, so the stall e2e could flake); the detector now times its own samples.
    • Under frontmcp dev, the implied plugin replaced an ObservabilityPlugin.init() the server installed itself
      (same plugin class, so the second record won: its tracing and request logs were gone), and duplicated every call
      event when an app installed one; the implied plugin is now skipped in both cases.
    • A server run as ESM did not start with stalls on, nor under frontmcp dev with @frontmcp/observability
      installed (logger.child is not a function): the FrontMcpLogger class CallMonitor injected belonged to the
      plugin's CommonJS copy of the SDK. Checked by hand with an .mts server run by tsx: it now starts and writes
      its call events.
    • Each server of a splitByApp server, or of a standalone app, ran its own detector: every stall was reported once
      per server, the others saying with no call in flight, and gc and unknown were counted once per server. The
      detector is now one per process.
  • Run:
    • Unit: @frontmcp/sdk 7644, frontmcp (cli) 2092, @frontmcp/utils 2000, @frontmcp/observability 556,
      @frontmcp/skills 249, all passing.
    • E2E: nx test demo-e2e-standalone 54/54, nx test demo-e2e-cli-exec 201/201, nx test demo-e2e-dts-consumer
      17/17.
    • Lint on the touched projects (no new warnings), scripts/fix-unused-imports.mjs, prettier, and tsc --noEmit on
      the lib and spec tsconfigs of sdk, observability, cli and utils.

Summary by CodeRabbit

  • New Features
    • Added optional event-loop stall detection, with warnings that identify calls active during a stall and metrics that track stall duration.
    • frontmcp dev now records call outcomes and enabled stall events to .frontmcp/dev-events.jsonl by default. Use --events-file to choose another location or --no-events to disable recording.
    • Development events are also supported in --stdio --serve mode and can be configured with custom sinks.
  • Documentation
    • Added guidance and examples for enabling stall detection and recording development events.

GLOBAL providers can implement onBootstrap(), onReady() and onShutdown().
onBootstrap() runs in dependency order before anything is enumerated and a
throw fails the start with ProviderBootstrapError. onReady() runs once the
server serves, on every entry point. onShutdown() runs in reverse order when a
scope is disposed, each within lifecycle.shutdownHookTimeoutMs (default 3000 ms).

createContainer({ providers, apps }) builds the provider graph of a server
without its registries, flows or transport, and returns get, tryGet and
dispose.
… stdio scope

A second caller of runReadyHooks() or runShutdownHooks() returned while the
first run was still going, so a concurrent connect() could return before
onReady() finished. The stdio SIGTERM handler ran onShutdown() for the primary
scope only, so a standalone app's providers were readied but never shut down.
…) providers the config passed

- @frontmcp/edge builds its server with createForGraph() and never ran the providers' onReady(); the
  worker and each session Durable Object now run them once, when their first request builds the server.
  FrontMcpInstance.emitProvidersReady() is public so a host built from createForGraph() can run them.
- createContainer() registered FrontMcpConfig as { logging } only, so a provider that reads the config
  (info, redis via getGlobalStoreConfig) failed or saw nothing; it now gets the object passed.
- The lifecycle e2e tolerates reading the journal while it is rewritten, and the stdio unit spec waits
  for the exit instead of a fixed delay.
- create-provider: onReady() is awaited before the entry points return and the signal handlers are
  installed; the name match covers third-party values; hooks run once per server; edge and FrontMcpConfig.
- SqliteStorageAdapter: conditional set, delete, incr, deleteIfEquals and
  expireIfEquals each run in one BEGIN IMMEDIATE transaction, so they are
  atomic across processes; expire and expireIfEquals validate the TTL and
  treat an expired key as missing, like the memory and Redis adapters.
  DistributedSemaphore now works on SQLite.
- One connection per database file per process, shared by every store on
  the file, with a queue that runs the adapter's operations in order and
  waits for another process's lock on a timer instead of blocking the
  event loop.
- createStorage({ type: 'sqlite', sqlite: { path } }) loads
  @frontmcp/storage-sqlite on demand; an invalid sqlite block throws
  StorageConfigError whatever the fallback.
- SDK: sqliteStorageProvider() and SqliteStorageToken give providers and
  tools a SQLite-backed storage, disconnected at shutdown.
- New e2e app demo-e2e-local-runtime: ten processes contend on one file.
…-provider-lifecycle

# Conflicts:
#	libs/sdk/src/front-mcp/front-mcp.ts
…cope that holds it

Under splitByApp, a server-level useValue provider is held by every app scope,
and the first scope to build it ran its onShutdown() when that scope was
disposed, before the providers of later scopes that depend on it. Each
registry of the tree now counts as a holder: the first runs the bootstrap and
ready hooks, and the last to shut down runs onShutdown().
…ue the expiry sweeps

- A queued operation's busy timeout counts from its call, restarting when an earlier queued
  operation gets the lock, so a held lock no longer makes queued operations time out one after another.
- SqliteStorageAdapter.disconnect() lets the operations already called finish before closing the file.
- The KV and event stores run their periodic sweeps on the connection's queue: no event-loop stall,
  and no SQLITE_BUSY thrown from the timer while another process holds the lock.
- A store whose setup fails releases the shared connection, and a handle closed directly is replaced
  for the stores opened after it.
A registry that held the same hook-bearing value under two tokens released it
at its second token, so onShutdown() ran on it before the providers built
between the two tokens that depend on it. A registry now holds each instance
once, at its first token.

Also complete the multi-instance-cleanup example's feature sentence.
…he server down

When one app failed while another was still running a provider's onBootstrap(),
the server shut down at once and that provider finished bootstrapping after it,
so its onShutdown() never ran. The app registry now waits for every app to
settle before it reports the first failure.
ping() queried the database directly with the full busy timeout, so a commit
from another process in rollback-journal mode blocked the event loop for up
to busyTimeoutMs. It now waits on the queue with timers like every other
operation.

Also: the e2e worker helper resolves on close so the last stdout lines are
read; the lock example takes a unique token and releases it; the createStorage
lock snippets set fallback: 'error'; WAL backups use VACUUM INTO.
…n when orphaned

Move the CLI's process-tree helpers into a Node-only process module in
@frontmcp/utils, with spawnOwned, reapOrphans, watchParent and watchPath.
runStdio shuts down through the normal shutdown path when stdin ends or
the parent exits (lifecycle.exitWhenOrphaned, on by default for stdio,
opt-in for the HTTP and Unix socket servers). CliTaskRunner spawns its
workers owned under TASK_WORKER_MARKER so orphaned ones can be reaped.
…/m03-process-kit

# Conflicts:
#	libs/skills/catalog/frontmcp-development/references/create-provider.md
…tstrap

Sibling registries that hold the same value shared a holder count but not its
onBootstrap() run, so a registry that was not the first holder built and
bootstrapped the providers that depend on the value before the value finished
bootstrapping. The tree now shares the bootstrap run, and every holder awaits
it, so a failure reaches the dependents too.
…ke @frontmcp/utils

The lazy import('@frontmcp/utils') in Scope kept every export of the
barrel in an app's browser bundle, the new process helpers included.
mkdir already loads fs on first use, so a static import is enough.
runStdio attaches the watcher after the providers' onReady() hooks, so a
client that closed stdin meanwhile left the server running. Also document
the orphan behaviour on runStdio and runUnixSocket.
get() read an expired row and then deleted the key unconditionally, outside a
transaction, so a value another process wrote between the two statements was
deleted. The purge now deletes the row only if it is still expired.

Also: the WAL troubleshooting row no longer suggests walMode: false for an
existing WAL file.
…opped, and stop a child without its own group

- reapOrphans never stops the calling process, its ancestors or the leader of its process group. A task worker
  started through `npx tsx` carries the owner tag on its launchers too, so a worker that reaped after its server
  died stopped its own launchers and their group.
- A stopped orphan that its new parent does not collect (a zombie, as under a container's PID 1 without an init)
  counts as stopped: `kill(pid, 0)` still reaches a zombie, so the reaper waited out the grace period, sent SIGKILL
  and reported nothing. `ps` now reads the `stat` column.
- OwnedProcess.stop() on a child spawned without `detached` signals the child itself and kills it after the grace
  period. It resolved `true` at once while the child was still running, since such a child leads no process group.
…before shutting down

A stdio client that writes its requests and closes stdin (`cat requests.jsonl | node dist/main.js`) lost the reply
to any request still running: the orphan watch started the shutdown as soon as stdin ended, and closed the server
under it. Now `exitOnShutdownSignals()` tells the shutdown what triggered it, and on `stdin-ended` `runStdio()`
first waits, for up to 2 seconds (`STDIO_DRAIN_MS`), until the stdio router has written every reply it owes
(`StdioRevisionRouter.whenIdle()`; a cancelled request and a `subscriptions/listen` stream owe none). A signal
also stops the orphan watch, so a shutdown already under way logs no second reason.
…till expired

ttl() had the same read-then-delete sequence as get(): a value another process
wrote between the two statements was deleted. It now uses the guarded purge.
…alone under frontmcp dev

Under frontmcp dev, a server without an observability config got
ObservabilityPlugin.init({ tracing: false, logging: false }) for the dev events
even when it installed the plugin itself. Both records share the plugin class as
their token, so the second replaced the server's own (its tracing and request
logs were gone); a plugin installed on an app stayed, and every call event was
written twice. The events-only plugin is now added only when no plugin named
observability is installed on the server or an app.
…ge's module and harden the offload pool

- A worker error is public only when it is a FrontMCP McpError (by its class brand) with isPublic, as on the main
  thread; an error that only has an isPublic property no longer reaches the client in production. A public one keeps
  its JSON-RPC code, data and remedy, so a ResourceNotFoundError read in a worker answers -32002 with data.uri.
- The worker calls only a function its module exports, never one the exports inherit (toString, call).
- captureCallerFile no longer skips node_modules frames: a worker resource declared in an installed package is
  recorded as that module, not the module that required the package (CommonJS), and is found at all under ESM.
- Worker resources are keyed by module, name and URI or URI template, so two of one module or bundle that share a
  name are not read as each other.
- A worker's stdout is forwarded chunk by chunk instead of piped, so eleven or more workers no longer trip a
  MaxListenersExceededWarning on process.stderr.
- A tsx without the cjs/esm registration APIs is skipped in the worker instead of failing every offloaded call.
- The read flow loads its worker helper on demand, which takes about 570 bytes off the browser bundle.
- The refusal for a resource with no module on the stack says a bundle that inlines @frontmcp/sdk has none.
- Catalog: which errors stay public, installed packages and inlined-SDK bundles, preloads in the server's flags,
  N x maxWorkers threads with splitByApp, the tsx fallback, and a troubleshooting row.
…m11-worker-offload

# Conflicts:
#	apps/e2e/demo-e2e-dts-consumer/fixture/tsconfig.json
#	libs/sdk/src/common/metadata/front-mcp.metadata.ts
#	libs/sdk/src/common/tokens/front-mcp.tokens.ts
#	libs/sdk/src/common/types/options/index.ts
…lls to it

appendFile's mode applies only when it creates the file, so a recording path
that already existed kept whatever permissions it had, and recorded calls went
into a file others could read. The writer now sets an existing file to 0600
before its first line; a file it cannot do that for is not written to.

Also trim slashes in the testing client and transports with trimSlashes and
trimTrailing from @frontmcp/utils instead of regexes that backtrack
polynomially on runs of '/' (CodeQL js/polynomial-redos).
…at/m10-stall-dev-events

# Conflicts:
#	apps/e2e/demo-e2e-dts-consumer/e2e/dts-consumer.e2e.spec.ts
#	apps/e2e/demo-e2e-dts-consumer/fixture/tsconfig.json
#	libs/utils/src/fs/fs.ts
@frontegg-david
frontegg-david marked this pull request as ready for review November 2, 2026 16:50
@frontegg-david
frontegg-david added this pull request to stack #997 November 2, 2026 16:50
@frontegg-david

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Next included review available in 27 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available. Your 53 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between e774879 and 4667cce.

Files selected for processing: 62
  • apps/e2e/demo-e2e-cli-exec/e2e/cli-dev-events.e2e.spec.ts
  • apps/e2e/demo-e2e-cli-exec/project.json
  • apps/e2e/demo-e2e-dts-consumer/e2e/dts-consumer.e2e.spec.ts
  • apps/e2e/demo-e2e-dts-consumer/fixture/stalls-dev-events-must-compile.ts
  • apps/e2e/demo-e2e-dts-consumer/fixture/tsconfig.json
  • apps/e2e/demo-e2e-standalone/e2e/stall-attribution.e2e.spec.ts
  • apps/e2e/demo-e2e-standalone/project.json
  • apps/e2e/demo-e2e-standalone/src/apps/stall-probe/index.ts
  • apps/e2e/demo-e2e-standalone/src/apps/stall-probe/tools/busy-wait.tool.ts
  • apps/e2e/demo-e2e-standalone/src/apps/stall-probe/tools/fail-fast.tool.ts
  • apps/e2e/demo-e2e-standalone/src/main-stalls.ts
  • libs/cli/src/commands/dev/__tests__/dev-events-file.spec.ts
  • libs/cli/src/commands/dev/__tests__/resolve-dev-launch.spec.ts
  • libs/cli/src/commands/dev/bridge/__tests__/upstream-client.spec.ts
  • libs/cli/src/commands/dev/bridge/index.ts
  • libs/cli/src/commands/dev/bridge/upstream-client.ts
  • libs/cli/src/commands/dev/dev-events-file.ts
  • libs/cli/src/commands/dev/dev.ts
  • libs/cli/src/commands/dev/register.ts
  • libs/cli/src/core/__tests__/bridge.spec.ts
  • libs/cli/src/core/args.ts
  • libs/cli/src/core/bridge.ts
  • libs/observability/src/__tests__/call-monitor.spec.ts
  • libs/observability/src/__tests__/dev-events-server.spec.ts
  • libs/observability/src/__tests__/dev-events.spec.ts
  • libs/observability/src/__tests__/in-flight-calls.spec.ts
  • libs/observability/src/__tests__/observability-options.spec.ts
  • libs/observability/src/__tests__/stall-detector.spec.ts
  • libs/observability/src/__tests__/stall-report.spec.ts
  • libs/observability/src/dev-events/dev-event-stream.ts
  • libs/observability/src/dev-events/index.ts
  • libs/observability/src/index.ts
  • libs/observability/src/plugin/index.ts
  • libs/observability/src/plugin/observability.errors.ts
  • libs/observability/src/plugin/observability.options.ts
  • libs/observability/src/plugin/observability.plugin.ts
  • libs/observability/src/plugin/observability.plugin.types.ts
  • libs/observability/src/stalls/call-monitor.ts
  • libs/observability/src/stalls/in-flight-calls.ts
  • libs/observability/src/stalls/index.ts
  • libs/observability/src/stalls/stall-detector.ts
  • libs/observability/src/stalls/stall-report.ts
  • libs/sdk/src/builtin/call-recorder/__tests__/call-recorder.plugin.spec.ts
  • libs/sdk/src/common/interfaces/logger.interface.ts
  • libs/sdk/src/common/types/options/observability/interfaces.ts
  • libs/sdk/src/common/types/options/observability/schema.ts
  • libs/sdk/src/logger/__tests__/logger-token.spec.ts
  • libs/sdk/src/logger/logger.registry.ts
  • libs/sdk/src/metrics/__tests__/metrics.service.spec.ts
  • libs/sdk/src/metrics/metrics.service.ts
  • libs/sdk/src/scope/__tests__/dev-events-observability.spec.ts
  • libs/sdk/src/scope/__tests__/observability-scope.helper.spec.ts
  • libs/sdk/src/scope/dev-events-observability.ts
  • libs/sdk/src/scope/observability-scope.helper.ts
  • libs/sdk/src/scope/scope.instance.ts
  • libs/skills/catalog/frontmcp-development/references/create-plugin.md
  • libs/skills/catalog/frontmcp-observability/SKILL.md
  • libs/skills/catalog/frontmcp-observability/examples/stall-detection/find-a-blocking-tool.md
  • libs/skills/catalog/frontmcp-observability/references/metrics-endpoint.md
  • libs/skills/catalog/frontmcp-observability/references/stall-detection.md
  • libs/skills/catalog/frontmcp-setup/references/setup-project.md
  • libs/skills/catalog/skills-manifest.json
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between cba1ce0 and e774879.

Files selected for processing: 62
  • apps/e2e/demo-e2e-cli-exec/e2e/cli-dev-events.e2e.spec.ts
  • apps/e2e/demo-e2e-cli-exec/project.json
  • apps/e2e/demo-e2e-dts-consumer/e2e/dts-consumer.e2e.spec.ts
  • apps/e2e/demo-e2e-dts-consumer/fixture/stalls-dev-events-must-compile.ts
  • apps/e2e/demo-e2e-dts-consumer/fixture/tsconfig.json
  • apps/e2e/demo-e2e-standalone/e2e/stall-attribution.e2e.spec.ts
  • apps/e2e/demo-e2e-standalone/project.json
  • apps/e2e/demo-e2e-standalone/src/apps/stall-probe/index.ts
  • apps/e2e/demo-e2e-standalone/src/apps/stall-probe/tools/busy-wait.tool.ts
  • apps/e2e/demo-e2e-standalone/src/apps/stall-probe/tools/fail-fast.tool.ts
  • apps/e2e/demo-e2e-standalone/src/main-stalls.ts
  • libs/cli/src/commands/dev/__tests__/dev-events-file.spec.ts
  • libs/cli/src/commands/dev/__tests__/resolve-dev-launch.spec.ts
  • libs/cli/src/commands/dev/bridge/__tests__/upstream-client.spec.ts
  • libs/cli/src/commands/dev/bridge/index.ts
  • libs/cli/src/commands/dev/bridge/upstream-client.ts
  • libs/cli/src/commands/dev/dev-events-file.ts
  • libs/cli/src/commands/dev/dev.ts
  • libs/cli/src/commands/dev/register.ts
  • libs/cli/src/core/__tests__/bridge.spec.ts
  • libs/cli/src/core/args.ts
  • libs/cli/src/core/bridge.ts
  • libs/observability/src/__tests__/call-monitor.spec.ts
  • libs/observability/src/__tests__/dev-events-server.spec.ts
  • libs/observability/src/__tests__/dev-events.spec.ts
  • libs/observability/src/__tests__/in-flight-calls.spec.ts
  • libs/observability/src/__tests__/observability-options.spec.ts
  • libs/observability/src/__tests__/stall-detector.spec.ts
  • libs/observability/src/__tests__/stall-report.spec.ts
  • libs/observability/src/dev-events/dev-event-stream.ts
  • libs/observability/src/dev-events/index.ts
  • libs/observability/src/index.ts
  • libs/observability/src/plugin/index.ts
  • libs/observability/src/plugin/observability.errors.ts
  • libs/observability/src/plugin/observability.options.ts
  • libs/observability/src/plugin/observability.plugin.ts
  • libs/observability/src/plugin/observability.plugin.types.ts
  • libs/observability/src/stalls/call-monitor.ts
  • libs/observability/src/stalls/in-flight-calls.ts
  • libs/observability/src/stalls/index.ts
  • libs/observability/src/stalls/stall-detector.ts
  • libs/observability/src/stalls/stall-report.ts
  • libs/sdk/src/builtin/call-recorder/__tests__/call-recorder.plugin.spec.ts
  • libs/sdk/src/common/interfaces/logger.interface.ts
  • libs/sdk/src/common/types/options/observability/interfaces.ts
  • libs/sdk/src/common/types/options/observability/schema.ts
  • libs/sdk/src/logger/__tests__/logger-token.spec.ts
  • libs/sdk/src/logger/logger.registry.ts
  • libs/sdk/src/metrics/__tests__/metrics.service.spec.ts
  • libs/sdk/src/metrics/metrics.service.ts
  • libs/sdk/src/scope/__tests__/dev-events-observability.spec.ts
  • libs/sdk/src/scope/__tests__/observability-scope.helper.spec.ts
  • libs/sdk/src/scope/dev-events-observability.ts
  • libs/sdk/src/scope/observability-scope.helper.ts
  • libs/sdk/src/scope/scope.instance.ts
  • libs/skills/catalog/frontmcp-development/references/create-plugin.md
  • libs/skills/catalog/frontmcp-observability/SKILL.md
  • libs/skills/catalog/frontmcp-observability/examples/stall-detection/find-a-blocking-tool.md
  • libs/skills/catalog/frontmcp-observability/references/metrics-endpoint.md
  • libs/skills/catalog/frontmcp-observability/references/stall-detection.md
  • libs/skills/catalog/frontmcp-setup/references/setup-project.md
  • libs/skills/catalog/skills-manifest.json

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Walkthrough

This change adds event-loop stall detection and development-event recording to observability. It adds SDK configuration and call tracking, CLI JSONL and IPC delivery, integration tests, and documentation. It also updates call recorder tests to wait for conditions instead of fixed delays.

Changes: Observability and Development Events

Layer / File(s) Summary
Observability configuration and activation
libs/sdk/src/common/types/options/observability/*, libs/observability/src/plugin/*, libs/sdk/src/scope/*, libs/sdk/src/common/interfaces/logger.interface.ts, libs/sdk/src/logger/*, apps/e2e/demo-e2e-dts-consumer/*, libs/skills/catalog/frontmcp-development/references/create-plugin.md
Adds stall and development-event options, validation, plugin activation, and the shared FRONTMCP_LOGGER token. Scope initialization can infer observability configuration from development-event environment settings when no explicit configuration exists.
Call tracking and stall reports
libs/observability/src/stalls/*, libs/observability/src/__tests__/{call-monitor,in-flight-calls,stall-detector,stall-report}.spec.ts, libs/sdk/src/metrics/*
Tracks in-flight calls and samples event-loop gaps. Reports include call attribution and GC time; the event-loop stall counter is included in the process metrics category.
Development-event stream and sinks
libs/observability/src/dev-events/*, libs/observability/src/__tests__/{dev-events,dev-events-server}.spec.ts
Adds event batching, queue limits, JSONL and IPC sinks, and shutdown handling. Server tests cover event output with default, server-installed, and app-installed observability plugins.
CLI event-file and IPC delivery
libs/cli/src/commands/dev/*, libs/cli/src/core/{args,bridge}.ts, libs/cli/src/commands/dev/**/__tests__/*
Adds --events-file and --no-events, initializes the selected file, and forwards child events from pipe-mode IPC to JSONL output.
End-to-end verification and documentation
apps/e2e/demo-e2e-cli-exec/*, apps/e2e/demo-e2e-standalone/*, libs/skills/catalog/frontmcp-observability/*, libs/skills/catalog/frontmcp-setup/references/setup-project.md, libs/skills/catalog/skills-manifest.json
Adds E2E coverage for call outcomes, stall attribution, event deduplication, and stdio bridge delivery. Adds stall-detection references and setup guidance.

Call Recorder Test Synchronization

Layer / File(s) Summary
Condition-based test waits
libs/sdk/src/builtin/call-recorder/__tests__/call-recorder.plugin.spec.ts
Replaces fixed delays and polling with waits for task completion, complete file lines, and warning output.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DevCLI
  participant ChildServer
  participant CallMonitor
  participant DevEventStream
  participant PipeBridge
  participant EventsFile
  DevCLI->>ChildServer: configure development-event output
  ChildServer->>CallMonitor: execute call hooks
  CallMonitor->>DevEventStream: emit call or stall event
  DevEventStream->>PipeBridge: send event batch over IPC
  PipeBridge->>EventsFile: append JSONL event lines
Loading
Merge Risk: Moderate · previous review e7748

Resolve the unbounded event-write backlog before merging. The startup, shutdown-event, test-reliability, and contract issues also warrant fixes or explicit acceptance.

Pre-merge checks: Passed 4 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Docstring Coverage Inconclusive Docstring coverage is 35.48% which is insufficient. The required threshold is 65.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly summarizes the two main changes: stall call attribution and development-event streaming from frontmcp dev.
Description check Passed The description is complete and relevant. It explains the motivation, implementation, user-facing changes, documentation updates, and extensive unit and E2E validation.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 35.48% which is insufficient. The required threshold is 65.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 31 functions across 50 files. (12 skipped: 10 unsupported, 2 over the file limit.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
Generate docstrings
  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Performance Test Results

Status: ✅ All tests passed

Summary

Project Tests Passed Warnings Failed Leaks
✅ demo-e2e-agents 4 4 0 0 0
✅ demo-e2e-cache 11 11 0 0 0
✅ demo-e2e-codecall 4 4 0 0 0
✅ demo-e2e-config 4 4 0 0 0
✅ demo-e2e-direct 3 3 0 0 0
✅ demo-e2e-elicitation 1 1 0 0 0
✅ demo-e2e-errors 4 4 0 0 0
✅ demo-e2e-hooks 3 3 0 0 0
✅ demo-e2e-multiapp 4 4 0 0 0
✅ demo-e2e-notifications 3 3 0 0 0
✅ demo-e2e-openapi 2 2 0 0 0
✅ demo-e2e-providers 4 4 0 0 0
✅ demo-e2e-public 4 4 0 0 0
✅ demo-e2e-redis 15 15 0 0 0
✅ demo-e2e-remember 4 4 0 0 0
✅ demo-e2e-remote 5 5 0 0 0
✅ demo-e2e-serverless 2 2 0 0 0
✅ demo-e2e-skills 15 15 0 0 0
✅ demo-e2e-standalone 2 2 0 0 0
✅ demo-e2e-transport-recreation 3 3 0 0 0
✅ demo-e2e-ui 4 4 0 0 0

Total: 101 tests across 21 projects

📊 View full report in workflow run


Generated at: 2026-11-02T18:09:10.233Z
Commit: 154e25e2

Base automatically changed from feat/m11-worker-offload to main November 2, 2026 17:07
…ents

# Conflicts:
#	apps/e2e/demo-e2e-dts-consumer/e2e/dts-consumer.e2e.spec.ts
#	apps/e2e/demo-e2e-dts-consumer/fixture/tsconfig.json
#	libs/skills/catalog/frontmcp-observability/references/metrics-endpoint.md
…sleeps

Test-only fix. The maxBytes spec waited a fixed 50 ms for the first queued
write before reading the file, and failed with ENOENT under a loaded parallel
run. The call recorder spec now polls (every 10 ms, up to 5 s) for the
condition each step needs: the file's first complete line, the maxBytes
warning, and the task leaving the working status.
@frontegg-david

Copy link
Copy Markdown
Contributor Author

Merged main (cba1ce0, the merged stack through #1009) into this branch in 757825c. main's tree equals 966f070, which this branch already contained, so the three conflicts (dts-consumer.e2e.spec.ts, fixture/tsconfig.json, metrics-endpoint.md) resolve to this branch's version, which holds both sides; the merged tree is the branch's tree.

e774879 is a test-only fix for a flake in call-recorder.plugin.spec.ts (on main since #1006): the maxBytes test waited a fixed 50 ms for the first queued write and failed with ENOENT under a loaded parallel run. The spec now polls, every 10 ms for up to 5 s, for what each step needs.

Local runs on e774879 (sdk and observability built with --skip-nx-cache): nx test sdk 602 suites / 7861 tests, nx test observability 546, demo-e2e-standalone 54, demo-e2e-cli-exec 201, demo-e2e-dts-consumer 23; the call-recorder spec passed five runs in a row under load.

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

Comment thread libs/cli/src/commands/dev/dev.ts Resolved
…heir file cannot be made, and keep events accepted during shutdown

frontmcp dev (and its stdio bridge) failed to start when the events file could
not be created, for instance with .frontmcp not writable. It now prints a
warning and starts with the server's events turned off.

DevEventStream.onShutdown() waited for one drain only, so an event accepted
after that drain emptied the queue, but before the stream closed, started a
drain the sinks were closed under and was lost. Shutdown now waits until no
drain is left before it closes the sinks.

Also: the dev-events server spec polls for the first call event instead of a
fixed 50 ms wait, the stall e2e allows the 1 ms that whole-millisecond clocks
can lose, and the stall docs say there is one detector per thresholdMs.
@frontegg-david
frontegg-david marked this pull request as ready for review November 2, 2026 17:53
@frontegg-david

Copy link
Copy Markdown
Contributor Author

Merged main (91531eb, the perf stack) into this branch in 4667cce, cleanly. The bundle measures 1,322,865 (create() + one tool) and 1,330,054 (+ WebMcpPlugin), within main's budgets (1,323,819 / 1,330,696); this PR raises no budget.

Local runs on 4667cce (sdk and observability built with --skip-nx-cache): sdk 8484, utils 2025, observability 547, cli 2103, skills 249; e2e dts-consumer 23, standalone 54, cli-exec 201, bundle-size 12.

frontegg-david added a commit that referenced this pull request Nov 2, 2026
On main's budgets (1,323,819 and 1,330,696), duties grow create() + one tool
by 1,309 bytes and create() + WebMcpPlugin + one tool by 1,308, measured
against #1013's merged head; the budgets move up by exactly that.
@alexmercerpo
alexmercerpo merged commit 0c3f110 into main Nov 3, 2026
46 checks passed
@alexmercerpo
alexmercerpo deleted the feat/m10-stall-dev-events branch November 3, 2026 07:27
alexmercerpo pushed a commit that referenced this pull request Nov 3, 2026
…ne or workspace runs (#1014)

* feat(sdk): add @duty, recurring work that one process runs under a lease, with takeover when the holder dies

A @duty({ every, scope: 'machine' | 'workspace', leaseMs }) provider has run() called every interval in the one
process that holds its lease. The lease is a store key with a TTL: set-if-absent to acquire, compare-and-expire to
renew every third of the lease and again before each run, compare-and-delete to release at shutdown. Expiry follows
the store's TTL, so another process takes over once a dead or paused holder stops renewing, and a holder that wakes
after a takeover never runs again. Leases live in a machine-wide SQLite file, or the server's Redis when configured.
Duties start on onReady and stop on shutdown; createContainer() builds them but runs them only with runDuties.

* fix(sdk): work out a duty's lease key before opening its store, so a failed key leaves no handle open

* fix(sdk): give a duty's lease up once it can't be renewed for a whole lease, and keep it while a stopped run winds down

A holder whose store calls kept failing (Redis unreachable, a locked SQLite file, a hung call) kept holding the duty
and its run in flight without a confirmed lease, so once the store ended the lease another process could run it at
the same time. The holder now arms a monotonic deadline from the moment it sent each successful acquire or renew;
past it, it aborts the run in flight, logs that it could not renew within leaseMs and contends again. Acquire
anchors that deadline with compare-and-expire instead of reading the key back.

stop() stopped renewing before the aborted run returned, so a run that took longer than the rest of the lease to
wind down overlapped the next holder's. It now keeps renewing until the run returns, then releases the lease.

A takeover waited the last-run marker's TTL rounded up to whole seconds, which for an interval near the timer limit
overflowed setTimeout and ran at once; it now waits at most one interval. A process that doesn't get the lease logs
once that the duty already runs elsewhere, with the lease key.

* fix(sdk): keep duty leases apart and owner-only, and run a container duty's onReady() before it starts

- Two duty classes with the same lease in one process (the same name in one server, or two plugins' classes of the
  same name) left one never running; the second to start now fails with DutyNameConflictError. The same class built
  more than once (per scope of a splitByApp server) still shares one lease.
- Lease keys URI-encode the server and duty names, so a ':' in either can't make two leases one or hit the
  last-run marker.
- createContainer({ runDuties }) needs info.name: without it every unnamed container on the machine shared the
  'frontmcp' leases. It now runs each duty's own onReady() before starting it, as a server does.
- The lease folder is created 0700 and duties.sqlite 0600 (with an append open, so processes starting together never
  truncate it); a folder another user owns, or the group or others can write to, or one above it that is not sticky,
  is refused with UnsafeDutyStoreError. The walk is the Unix-socket guard's, exported as findReplaceableDirectory().
  @frontmcp/utils writeFile() takes a flag option for the append open.
- The create-duty and create-provider references describe the conflict, the required info, file safety, giving a
  lease up, clocks, Redis hostnames, splitByApp and the new log lines.

* test(e2e): fit the browser bundle budgets to duties

@duty's decorator and option schema reach the browser bundle (the runner itself stays out), so create() + one tool grows by 1,309 bytes minified and create() + WebMcpPlugin + one tool by 1,306. The two budgets move up by exactly that, keeping the headroom they had.

* test(e2e): fit the browser bundle budgets to duties on top of main's

On main's budgets (1,323,819 and 1,330,696), duties grow create() + one tool
by 1,309 bytes and create() + WebMcpPlugin + one tool by 1,308, measured
against #1013's merged head; the budgets move up by exactly that.

* fix(sdk): give a duty runner its holder id in the constructor

The cloudflare isolate-safety check (demo-e2e-cloudflare) refuses a
randomUUID() it reads as module-eval: the class field initializer of
DutyRunner.holder. The id is now set in the constructor, which runs at the
same time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants