Repository navigation
Tag native OAuth connections and failures in MCP analytics - #274
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
masnwilliams
left a comment
There was a problem hiding this comment.
LGTM. Native connections are tagged native_oauth, native refusals now leave a record, and the tests cover both. tsc --noEmit and the route, auth-context, and analytics suites pass locally.
Worth doing here or in a quick follow-up
- Say which stage the failure came from. A native token can now produce a
rejected/401native_oauthevent from two places: the issuer refusing it inauthenticateNativeOAuth, or/auth/contextrefusing the exchanged token inhandleMcpRequestWithIdentity. The two events are identical, but they're the two failures you'd most want to tell apart during a rollout. Astage: "authentication" | "scope"field would separate them. The comment onMcpConnectionScopeFailureAnalytics(and the matching test comment) also still says only scope resolution emits this event. Related: ClerkverifyTokenfailures emit nothing, so native and Clerk failure rates won't compare directly. Fine either way, but worth saying explicitly in the comment.
Optional
- The two capture sites build
upstreamStatusCodewith opposite ternaries for the same rule.NativeOAuthResult's failure arms already fitMcpConnectionContextFailure, socaptureMcpConnectionScopeFailurecould take{ stage, credentialType, failure }and readstatusCodein one place. I tried this locally and the production code typechecks unchanged; only thecapturedassertions need reshaping. authMethod: isApiKey ? "api_key" : credentialTypetakes its value from two sources. Gating oncredentialType === "api_key" ? isApiKey : isOAuthand returningauthMethod: credentialTypewould make the route's credential type the only source. It also gives the newnative_oauthassertion inauth-context.test.tssomething to check beyond the argument coming back unchanged.- The revocation test asserts
capturedafter the 401 but not after the 403 refused-scope case. That would be worth asserting too.
masnwilliams
left a comment
There was a problem hiding this comment.
lgtm, passing the credential type down from the route is the right call. a few things, none blocking:
1. please settle what upstream_status_code means before merge. for api keys it's the status the kernel api actually returned. for native failures it's the 401/403 that native-oauth.ts assigns itself, which also covers purely local refusals (failed local verification, oversized token, native oauth not configured), and the issuer's real http status on RFC 8693 errors is never recorded. once these events ship, any dashboard grouping by this field mixes the two meanings and the historical events can't be relabeled. either leave it unset for native and add something like response_status_code, or document it as "the status we returned" for every failure. if the latter is the intent, a comment is enough.
2. (suggestion) the capture + response block is now duplicated in route.ts. the native failure type already fits McpConnectionContextFailure, so one helper covers both paths:
// A refused connection never reaches the instrumented server, so this
// capture is its only analytics record.
function refuseConnection(
req: Request,
credentialType: McpCredentialType,
failure: McpConnectionContextFailure,
): Response {
captureMcpConnectionScopeFailure({
outcome: failure.status,
credentialType,
upstreamStatusCode: "statusCode" in failure ? failure.statusCode : undefined,
});
if (failure.status === "invalid")
throw new Error("Unable to resolve Kernel connection scope");
return connectionScopeFailureResponse(req, failure);
}then both call sites become if (x.status !== "ok") return refuseConnection(req, <type>, x);. tsc and the full suite pass with this locally.
3. (suggestion) connectionAnalyticsFromContext takes authMethod from two sources. "api key or not" comes from /auth/context, "which oauth" from the caller, and the new param is ignored for api-key contexts. making the route's credential type the only source and keeping the context check as a guard reads more directly:
if (!(credentialType === "api_key" ? isApiKey : isOAuth)) return null;
return { authMethod: credentialType, /* ... */ };only changes behavior when the two disagree (emits nothing instead of a mixed label); a test for the mismatched case would pin it.
nits
- the comment on
McpConnectionScopeFailureAnalyticsstill says the event is about scope resolution; it now also covers native authentication failures. observeConnectionsinroute.test.tsreads like the route'sobserveConnectionbut actually forcesisMcpAnalyticsEnabled; something likeforceAnalyticsEnabledwould be clearer.- clerk jwt verification failures are now the only oauth path with no failure event. fine for a follow-up.
Summary
MCP analytics could not tell Kernel-issued (native) OAuth apart from Clerk OAuth, and native authentication failures left no analytics record at all: they return 401/403/503 straight from the native path, before the scope resolution that emits
mcp_connection_scope_failure. That makes a native rollout hard to watch, since a native-only failure or drop in connections is invisible.$mcp_initializeevents now report$mcp_auth_method: "native_oauth"for native connections. Clerk OAuth stays"oauth"and API keys stay"api_key", so existing queries keep their meaning.mcp_connection_scope_failurewithconnection_credential_type: "native_oauth"and the outcome (rejectedorunavailable).mcp_connection_scope_failurenow records where it happened and what the client got:connection_failure_stage(authenticationfor this server refusing a native token,scopefor the Kernel API refusing the connection) andresponse_status_code(the status this server returned).upstream_status_codekeeps one meaning: the status the Kernel API returned, so it is set only forscopefailures. Clerk token verification failures are still not recorded.The credential type is passed from the route, which already knows which path authenticated the request;
/auth/contextreportsjwt/oauthfor both OAuth kinds.Tests
initializeproduces$mcp_auth_method: "native_oauth"; a revoked native grant, a refused scope and an issuer outage each capture anauthenticationfailure with the status returned; API-key refusals capturescopefailures with both statuses.connectionAnalyticsFromContextreports the route's credential type and nothing when it disagrees with/auth/context.bunx tsc --noEmitandbun test(1191 tests) pass.