Repository navigation
feat(sdk,testing,cli): record client calls and replay them against a new version - #1006
Merged
Merged
Conversation
…d replay them with replaySession() and frontmcp test --replay - CallRecorderPlugin (built into @frontmcp/sdk) appends each tools/call, resources/read and prompts/get a client makes, with the answer or error it received, to a redacted JSONL file from Finally hooks; writing never blocks or fails a call, and a write failure or a full file is logged once. - Redaction reuses the request log's credential names (now isCredentialName) and redactFields, moved to @frontmcp/utils with a key test, nested lists and a redactJson() that reaches into JSON text. - replaySession(file, server) in @frontmcp/testing replays a recording, one client per recorded session, and reports each call as same, changed, new-error or gone-error, ignoring volatile paths and folded result-handle ids; toMatchRecording() fails with a readable report. - frontmcp test --replay <file> [--replay-ignore <paths>] replays against the project server started through TestServer and exits 1 on a difference. - prompts:get-prompt keeps its answer in state.output, as the tool and resource flows do.
…report when a client fails to disconnect - The run that creates a task answers only its handle, so the recorder now records the task's own run, which answers; the call replays as a plain call instead of twice. - replaySession() disconnects its clients with allSettled, so a failed disconnect no longer hides the report. - The replay-calls reference says redaction goes by field name, never by value, and to review a recording before committing it.
… of call recordings, and create them owner-only - redactJson() also redacts, in text that is not JSON, a Bearer or Basic credential, a JWT, the user:password@ of a URL and the value given to a redacted name (?access_token=, apiKey: "...", TOKEN=...), in linear time - the recorder applies it to the input, the answer, a public error's message and a resource URI, redacts session id fields, and redacts a folded result-handles answer's handle wherever it appears, also in a later call reading it - a public error is recorded with the text its client reads (a tool error's remedy line included); an internal error only with its code - isCredentialName() matches a name as written as well as split into words, so passWord and X-PassWord are hidden again, and words ending in token, secret, password, passwd or apikey and the words jwt and passphrase match - the recording is created 0600, maxBytes counts the file's size on disk, and lines waiting for a slow disk are capped at maxBytes, dropping later calls with one warning - appendFile() takes the mode a new file gets
…a redacted text as a pattern - two errors with the same code differ at error.message when the recording kept a public error's message; error.message can be ignored, and an internal error is compared by code - a JSON-RPC error's message is read without its "MCP error <code>: " prefix - a [REDACTED] inside a recorded text matches any text there, so a value redacted inside a text, such as a folded answer's handle, is not a change; compareResultHandles is gone, as recordings no longer hold handles - a spec records real calls with CallRecorderPlugin and replays them, pinning the line format the replay reads - the replay-calls catalog entry documents the recording's redaction, error messages and file mode
…t/m09-call-replay # Conflicts: # apps/e2e/demo-e2e-dts-consumer/e2e/dts-consumer.e2e.spec.ts # apps/e2e/demo-e2e-dts-consumer/fixture/tsconfig.json
frontegg-david
marked this pull request as ready for review
November 2, 2026 16:10
Contributor
Author
|
@coderabbitai review |
Contributor
|
Warning Review limit reachedThis review is too large to run within your organization's remaining usage spending cap. Raise or remove your spending cap in the billing tab, then retry. Next included review available in 40 minutes. View limit details
 |
| Layer / File(s) | Summary |
|---|---|
Shared redaction and file utilities libs/utils/src/fs/*, libs/utils/src/redaction/*, libs/observability/src/logging/*, libs/sdk/src/scope/flows/request-log.redaction.ts, libs/sdk/src/scope/flows/__tests__/request-log.redaction.spec.ts, libs/skills/catalog/frontmcp-observability/references/structured-logging.md |
Adds shared field and text redaction utilities and appendFile. Observability now re-exports redaction from @frontmcp/utils. Request-log credential and session-ID name checks use normalized names. |
Recorder contract and capture pipeline libs/sdk/src/builtin/call-recorder/*, libs/sdk/src/errors/*, libs/sdk/src/index.ts, libs/sdk/src/prompt/flows/get-prompt.flow.ts, apps/e2e/demo-e2e-dts-consumer/* |
Adds recorder options and record types, a queued JSONL writer, and a plugin that records tool, resource, and prompt calls. Tests cover redaction, filtering, errors, file limits, and write failures. The strict declaration suite checks recorder types and options. |
Replay execution and comparison libs/testing/src/replay/*, libs/testing/src/matchers/*, libs/testing/src/errors/index.ts, libs/testing/src/index.ts, apps/e2e/demo-e2e-testing/e2e/call-replay.e2e.spec.ts, apps/e2e/demo-e2e-testing/src/apps/replay/*, apps/e2e/demo-e2e-testing/src/main.replay.ts |
Adds JSONL validation, ordered replay, response comparison, difference reporting, and the toMatchRecording matcher. End-to-end tests exercise changed answers, new and resolved errors, and ignored volatile fields. |
CLI replay and usage examples libs/cli/src/commands/dev/*, libs/cli/src/core/*, libs/skills/catalog/frontmcp-testing/*, libs/skills/catalog/frontmcp-setup/examples/project-structure-standalone/dev-workflow-commands.md, libs/skills/catalog/skills-manifest.json |
Adds frontmcp test --replay and --replay-ignore parsing and execution. Tests cover project testing-package resolution, server lifecycle, report output, and exit status. Skill references and examples document recording and replay. |
Estimated code review effort: 4 (Complex) | ~60 minutes
Sequence Diagram(s)
sequenceDiagram
participant McpTestClient
participant ReplayServer
participant CallRecorderPlugin
participant CallRecordWriter
participant JSONLFile
participant replaySession
participant compareCall
McpTestClient->>ReplayServer: Send tool, resource, or prompt call
ReplayServer->>CallRecorderPlugin: Run call flow hooks
CallRecorderPlugin->>CallRecordWriter: Write serialized call record
CallRecordWriter->>JSONLFile: Append JSONL line
replaySession->>JSONLFile: Read recorded calls
replaySession->>McpTestClient: Replay calls in recording order
McpTestClient->>ReplayServer: Send replay request
ReplayServer-->>McpTestClient: Return response
replaySession->>compareCall: Compare recorded and replayed results
Suggested reviewers: alexmercerpo
Merge Risk: 
d482c
-
Restrict permissions on existing recording files before merging to avoid exposing call data. The Windows test assertions and misplaced real-server spec also need correction.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 46.34% which is insufficient. The required threshold is 65.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Title check The title clearly and concisely summarizes the main change: recording client calls and replaying them against a newer server version. Description check The description includes the required What and why, User-facing change, and Tests sections. It explains the implementation, affected APIs and CLI options, documentation updates, test coverage, and val… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Full details: Description check
-
Explanation
The description includes the required What and why, User-facing change, and Tests sections. It explains the implementation, affected APIs and CLI options, documentation updates, test coverage, and validation results.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 46.34% which is insufficient. The required threshold is 65.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 41 functions across 46 files. (7 skipped: 7 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches
-
Generate docstrings
-
- Commit to this branch
- Create a new PR
Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
…t/m09-call-replay
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
Action not completed
|
Contributor
Performance Test ResultsStatus: ✅ All tests passed Summary
Total: 101 tests across 21 projects 📊 View full report in workflow run Generated at: 2026-11-02T16:44:50.607Z |
frontegg-david
marked this pull request as draft
November 2, 2026 16:30
…lls to it appendFile's mode applies only when it creates the file, so a recording path that already existed kept whatever permissions it had, and recorded calls went into a file others could read. The writer now sets an existing file to 0600 before its first line; a file it cannot do that for is not written to. Also trim slashes in the testing client and transports with trimSlashes and trimTrailing from @frontmcp/utils instead of regexes that backtrack polynomially on runs of '/' (CodeQL js/polynomial-redos).
frontegg-david
marked this pull request as ready for review
November 2, 2026 16:30
Contributor
Author
|
Also in f0f46db: CodeQL on d482cbd reported two Merged Local runs on f0f46db: SDK call-recorder suites 23/23, |
alexmercerpo
approved these changes
Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1005
What and why
Proposal item 5.3: record real calls against one version of a server and replay them against the next, so a
change to a tool, resource or prompt shows up as a diff of the answers clients would get, not as a surprise in
production.
The reported problem: a server's tests cover the cases its authors thought of, while clients send the calls they
actually send. There was no way to capture those calls and check a new build against them.
What this PR adds:
CallRecorderPlugin, built into@frontmcp/sdk(likeConfigPlugin). A plugin rather than a server option:recording is a set of hooks, it is opt-in with options of its own, it works on the server (every app) or on one
@App, and it needs no new@FrontMcpfield. Being in the SDK, it needs no extra install, so a production-likebuild can record without depending on
@frontmcp/testing.Will('parseInput')hooks start the clock, andFinallyhooks write the line, ontools:call-tool,resources:read-resourceandprompts:get-prompt. A call is written at theFinallyof the first answer stagethat sees a failure (an earlier stage's failure, or its own), else at the
Finallyof the last answer stage(
meterAnswer,fitAnswer,finalize). So the recorded answer is the one the client received: after every hook,the answer budget and shaping, and folded by the result-handles plugin when it folds.
v,at,method,name(the tool or prompt name the client asked for, or the URI),input,answerorerror,durationMs,session,client.erroris{ code, message }: the code theclient saw (a tool error answer's
_meta.code, or the JSON-RPC code, computed with the same functions the MCPhandlers use) and, for a public error, the text the client reads as a production server words it (a tool
error's text with its
To fix:remedy line; a JSON-RPC message without theMCP error <code>:prefix). Aninternal error is
{ code }only: its text is generic and carries a new error id each time.this.callTool()is not (ctx.isClientCall()). A run that endswith neither an answer nor an error (a hook's
handled()) writes no line. A task-augmented call is recordedonce, by the task's own run (the run that creates the task answers only a handle), so it replays as a plain call.
@frontmcp/utils(a new
appendFile, which takes amode), to a file created owner-only (0600). The writer flushes onshutdown (
onShutdown), a write failure is logged once, andmaxBytes(10 MiB) stops the file growing, loggedonce; it counts the file's size on disk, so lines another process appends to the same path count too. While
maxBytesof lines wait for a slow disk, later calls are dropped (logged once), so the queue never holds more.CallRecorderPlugin.init():path(.frontmcp/calls.jsonl),methods,redactFields,keepFields,maxBytes. Invalid options throwCallRecorderConfigurationErrornaming eachproblem.
isCredentialName(): a namematches as written (lowercased) or split into words (camelCase and snake_case), so
apiKey,access_token,clientSecretandpassWordmatch asx-api-keydoes; a word ending intoken,secret,password,passwdorapikey(accesstoken,x-amzn-oidc-accesstoken) and the wordsjwtandpassphrase(
x-goog-iap-jwt-assertion) now match too. Observability'sredactFields()moves to@frontmcp/utils, whereit takes a key test as well as names and walks nested lists;
redactJson()uses the same walker at any depth andinside text that holds JSON (a tool's text block repeats its
structuredContent), and in any other text redactsa
Bearer/Basiccredential, a JWT, theuser:password@of a URL, and the value given to a redacted name(
?access_token=…,#id_token=…,apiKey: "…",TOKEN=…). The recorder redacts the input, the answer, theerror message and the resource URI with
isCredentialName, session id names (sessionId), plusredactFields, minuskeepFields; it redacts the handle of a folded result-handles answer (its encryption keysderive from it) wherever it appears, also in a later call that passes it back; it records no headers and no
request
_meta, and cuts the session id to 8 characters withheaderValueForLog, as the request log does.@frontmcp/observabilitystill exportsredactFields: the same arguments work (it also takes a key test now),with the same depth limit (5) and return shape (the same object when no field is named).
replaySession(file, server, options)in@frontmcp/testing. Replays the calls in order, one client perrecorded session with its recorded client info, and returns
{ file, calls, differences }. Each call issame,changed(with each differing path, through JSON text too),new-errororgone-error; two errors compare bycode, then by message where the recording kept one (a public error's), so a reworded public error is
changedaterror.message.ignoretakes dot-path globs (*one key,**any number), anderror.message.A redacted value is not compared, and a
[REDACTED]inside a text matches any text there (so a folded answer'snew handle is no change).
formatReplayReport()prints the differences, and thetoMatchRecording()matcherfails with that report. A spec in
@frontmcp/testingrecords real calls withCallRecorderPluginand replaysthem, so the line format the replay reads stays pinned to the one the SDK writes.
frontmcp test --replay <file> [--replay-ignore <paths>]. Starts the project server throughTestServer(
npx tsx <entry>on a free port, the entry resolved asfrontmcp devdoes, with the.envand config overlaysof
frontmcp test), replays the file with the project's own@frontmcp/testing, prints the report and exits 1on a difference.
prompts:get-promptnow keeps its answer instate.output, as the tool and resource flows do, so aFinallyhook can read it.
User-facing change
CallRecorderPluginandCallRecorderConfigurationErrorin@frontmcp/sdk; newreplaySession(),formatReplayReport(),toMatchRecording()andReplayRecordingErrorin@frontmcp/testing;new
frontmcp test --replayand--replay-ignore;redactFields(),redactJson()andappendFile()in@frontmcp/utils;redactFields()also redacts objects inside nested lists; the debug request log hides morecredential headers.
CallRecorderPlugin.init({ path }), replaywith
replaySession()in a spec orfrontmcp test --replayin CI. The recorder matches field names by words, sobesides the request log's credential names it redacts
key,cacheKey,pageToken,nextPageToken,sessionIdand the like (list a safe one inkeepFields). Observability log redaction (redactFields): anobject in a list of lists was logged unredacted → it is redacted; nothing it redacted before is logged now.
Debug request log headers:
x_api_key-style names (underscores), camelCase names (X-AuthToken), words endingin
token/secret/password/passwd/apikey(x-amzn-oidc-accesstoken,x-csrftoken) and the wordsjwtand
passphrase(x-goog-iap-jwt-assertion) were logged → they are[REDACTED]; every header hidden beforestays hidden.
prompts:get-prompthooks:state.outputwas empty → it holds the answer.https://frontmcp.dev/reference/cli (
frontmcp test), https://frontmcp.dev/reference/server/observability (fieldredaction), and the plugins reference for the new built-in plugin.
frontmcp-testing(newreferences/replay-calls.mdandexamples/replay-calls/record-and-replay.md, a routing row and afrontmcp test --replayrow inSKILL.md),frontmcp-setup/examples/project-structure-standalone/dev-workflow-commands.md(frontmcp test --replay),frontmcp-observability/references/structured-logging.md(whereredactFieldslives, nested lists), andskills-manifest.json.Tests
@frontmcp/sdk:call-recorder.plugin.spec.ts(21): a tool call with input, answer, duration, session andclient; tool errors (a thrown public error, an output-schema refusal, an unknown tool) with their client codes; a
public error's text with its remedy line, no message for an internal error, and a JWT in a public message
redacted; a tool error the client gets as JSON-RPC (
-32601); resource reads and prompts, and a resource'sJSON-RPC code and message; a resource URI's
access_tokenredacted; the answer after a hook replaced it; noconfigured field or credential in the input, the answer or its JSON text; no credential under an innocuous key,
in a URL (userinfo, query), a JWT, a nested
headers.Authorizationor asessionId; no result handle in a foldedanswer or the later call reading it; the file created
0600; only the calls to the app it is installed on;keepFields;methods; in-process calls not recorded; a task-augmented call recorded once with the task'sanswer; no line for an unanswered run; a write failure logged once with the calls still answered;
maxByteslogged once, and counted on the file's size after another writer appended; invalid options.
call-record.writer.spec.ts(1): lines dropped whilemaxByteswait for a slow disk, logged once.request-log.redaction.spec.ts(36), includingpassWord,x_api_key,accesstokenand session id names.@frontmcp/testing:replay-diff.spec.ts(12),replay-session.spec.ts(8),mcp-matchers.replay.spec.ts(3),
replay-recording.spec.ts(1: real calls recorded byCallRecorderPluginhold only fields the replay knowsand replay unchanged).
@frontmcp/utils: the movedredaction.spec.tsplus 12 new cases (credentials in text,URLs and assignments), and
appendFilewith its mode. CLI:replay.spec.ts(6),test-replay.spec.ts, the--replay/--replay-ignoreparsing inprogram.spec.tsandbridge.spec.ts.apps/e2e/demo-e2e-testing/e2e/call-replay.e2e.spec.ts: records seven calls against fixture v1 (no password ortoken in the file); replayed against v2 the report names the changed
quote(structuredContent.price12 → 15),the new
payerror and the gonerefunderror; replayed against v1 it differs only on the volatile fields, and isclean once they are ignored.
demo-e2e-dts-consumer:call-recorder-must-compile.ts.nx run-many -t lint test build -p utils observability sdk testing cli skills(all green),nx test demo-e2e-testing(7 suites, 26 tests),demo-e2e-dts-consumer(17),demo-e2e-hooks(23),demo-e2e-standalone(53, for the request-log header rules), prettier on the changed files,scripts/fix-unused-imports.mjs. The redaction of text was run against 400 KB adversarial strings (long dotted,hyphenated,
{a:and quote runs) to check it stays linear.skills-manifest.jsonis the output oflibs/skills/scripts/generate-manifest.mjs(thebuildtarget runs itfirst and formats it with prettier), so the reflowed entries outside the testing skill are what the generator
writes; rerunning it leaves the file unchanged.
frontmcp test --replayon them: cleanand exit 0 with
--replay-ignore '**.quotedAt,**.quoteId', and exit 1 with thequotepaths listed without it.Summary by CodeRabbit