diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/.openspec.yaml b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/.openspec.yaml new file mode 100644 index 00000000000..cdcb5b60b5a --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/.openspec.yaml @@ -0,0 +1,3 @@ +schema: spec-driven +created: 2026-08-25 +status: draft diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/audit.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/audit.md new file mode 100644 index 00000000000..f8f890304d3 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/audit.md @@ -0,0 +1,36 @@ +## Audit Report: mcp-registry-provider + +**Last audited:** 2026-09-14T22:07:36Z + +### Summary + +| Category | CRITICAL | WARNING | SUGGESTION | +| -------- | -------- | ------- | ---------- | +| A | 0 | 0 | 2 | +| B | 0 | 0 | 0 | +| C | 0 | 4 | 0 | +| D | 0 | 0 | 2 | +| E | 0 | 0 | 0 | +| F | 0 | 0 | 0 | +| G | 0 | 1 | 1 | +| H | 0 | 0 | 0 | + +### CRITICAL + +- None + +### WARNING + +- **C** `openspec/changes/mcp-registry-provider/specs/mcp-registry-provider/spec.md:160` — `commit the complete set of produced entities to the catalog as a single **full** mutation`. Reconcile skip-on-map-failure with full-mutation membership. A skipped mapping omits the entity and will prune a still-listed registry server. Either retain last-good entities for accumulated servers that fail mapping, or state that a skip is an intentional prune. +- **C** `openspec/changes/mcp-registry-provider/specs/mcp-registry-provider/spec.md:64` — `perform an initial sync according to the schedule's initialDelay (or immediately when unset)`. Align design D3 and task 4.2 with this SHALL, or drop immediate-when-unset if `createScheduledTaskRunner` only honors `initialDelay`/`frequency`. Add a scenario for the chosen first-tick behavior. +- **C** `openspec/changes/mcp-registry-provider/design.md:76` — `a documented default (e.g. frequency: { minutes: 30 }, timeout: { minutes: 3 })`. Replace `e.g.` with the actual default `frequency`, `timeout`, and `initialDelay` (or explicitly unset) and copy those values into the spec omitted-schedule scenario and tasks 2.3/6.4. +- **C** `openspec/changes/mcp-registry-provider/design.md:126` — `ingested entities are pruned on the next catalog reconciliation because they are provider-managed via locationKey`. Rollback cannot emit a full mutation if the module is unregistered (absent config registers nothing). State orphan/retention behavior, or require an explicit empty mutation / catalog `orphanStrategy`. +- **G** `openspec/changes/mcp-registry-provider/design.md:68` — `each produced entity carries backstage.io/managed-by-location so it is visible in the catalog`. Fix the annotation value in D2 (for example `url:` or a `mcp-registry-provider:` location spec) and propagate that exact value into spec scenario `Provider attribution annotations present` and tasks 4.4/4.5. + +### SUGGESTION + +- **A** `openspec/changes/mcp-registry-provider/proposal.md:17` — `defaultOwner — the default spec.owner (a User/Group entity reference) applied to every produced API entity`. Mark `defaultOwner` optional (omit → mapping default `unknown`), matching design D1, the spec, and tasks 2.1/5.3. +- **A** `openspec/changes/mcp-registry-provider/tasks.md:17` — `schedule? (SchedulerServiceTaskScheduleDefinitionConfig)`. Keep the Config suffix only for `config.d.ts` and state it is the config-schema counterpart of `SchedulerServiceTaskScheduleDefinition` used in design D1, the proposal, and the spec; do not silently rename the runtime type. +- **D** `openspec/changes/mcp-registry-provider/.openspec.yaml:1` — `schema: spec-driven`. Set `schema: rhdh-spec-driven` to match `openspec/config.yaml`; this repo has no `openspec/schemas/spec-driven/`. +- **D** `openspec/changes/mcp-registry-provider/proposal.md:1` — `# Proposal: MCP Registry Provider`. Shorten the proposal to under 500 words per `openspec/config.yaml` (currently ~935 words); move pagination/API detail into design.md. +- **G** `openspec/changes/mcp-registry-provider/design.md:92` — `the provider calls the mapping transform, passing defaultOwner as the caller-override owner default`. Name the mapping export and caller-defaults object in D5 (prefix, owner, and whether lifecycle is omitted so mapping default `production` stands). Point task 5.1 at that signature; mapping D5 treats lifecycle as an ingestion-layer caller default this change never passes. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/design.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/design.md new file mode 100644 index 00000000000..f8ece8e240f --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/design.md @@ -0,0 +1,132 @@ +# Design: MCP Registry Provider + +## Canonical Touchpoints + +Carried forward from the proposal: + +- **PRDs (`specifications/prd/`)**: None +- **ADRs (`specifications/adr/`)**: None +- **Long-lived specs (`openspec/specs/`)**: None + +No canonical document updates. This change introduces a new capability only and does not modify any existing canonical document or long-lived spec. It **consumes** the sibling [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/design.md) transform contract without altering it. + +## Context + +[`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) defines a pure, deterministic `server.json` → `mcp-server` `API` entity transform and explicitly scopes out ingestion. This change supplies the runtime that calls that transform: a Backstage catalog **entity provider** that fetches servers from an [MCP Registry](https://github.com/modelcontextprotocol/registry) on a schedule and populates the catalog. + +**Reference prototype.** The [`mcp-reg-proxy-proto`](https://github.com/gabemontero/rhdh-plugins/tree/mcp-reg-proxy-proto) branch of `rhdh-plugins` (plugin at `workspaces/ai-integrations/plugins/mcp-registry-proxy-backend/`) is a _proxy_ — it re-exposes registry endpoints through RHDH. It is a **design reference only**; this change is a _provider_ (one-way ingestion into the catalog), not a proxy. Two lessons carry over: + +1. **Config shape.** The prototype placed config under `mcp.registry.proxy.` reading `baseUrl` and `registryVersion`. This change instead uses the idiomatic Backstage catalog-provider location `catalog.providers.mcpRegistry` as a **single object** (see D1). Multi-registry keyed maps are out of scope for the initial implementation; `baseName` is the forward-compatible hook for a later change. +2. **Pagination gap.** The prototype defined a `PaginatedResponse` type but **only fetched the first page** — it never followed `nextCursor`. This change treats full cursor traversal as a first-class requirement (see D4). + +**Registry API.** Per the [generic registry API](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/api/generic-registry-api.md#basic-example-list-servers), `GET //servers?cursor=&limit=` returns: + +```json +{ + "servers": [ { "server": { "name": "...", "description": "...", "version": "..." }, "_meta": { ... } } ], + "metadata": { "count": 10, "nextCursor": "..." } +} +``` + +Pagination is cursor-based: omit `cursor` on the first request; pass the prior `metadata.nextCursor` on each subsequent request; stop when it is absent, null, or empty. Cursors are opaque. + +**Backstage integration.** The plugin is a `catalog-backend-module` that registers one `EntityProvider` and schedules it via `SchedulerService`. On each tick it lists servers, maps them, and calls `connection.applyMutation({ type: 'full', entities })`. + +## Goals / Non-Goals + +**Goals:** + +- A scheduled catalog entity provider that ingests MCP servers from one configured registry into the RHDH catalog as `mcp-server` `API` entities. +- **Complete ingestion via cursor pagination** (the gap left by the reference prototype), capped per sync by optional `pageLimit` (default `10` pages), with optional `pageSize` controlling `?limit=`. +- Idiomatic, upstream-aligned configuration (`catalog.providers.mcpRegistry`) for a single registry, with optional `baseName` overriding the mapping identity prefix. +- Clean delegation to `mcp-registry-server-mapping` — the provider never reimplements the transform. +- Full-mutation semantics so the catalog converges to the registry's current state (adds, updates, prunes). +- Resilient, predictable (agent-native) error handling: skip a bad entry, fail a bad run without corrupting catalog state. + +**Non-Goals:** + +- The `server.json` → entity transform, annotation projection, secret redaction (owned by `mcp-registry-server-mapping`). +- A registry proxy / pass-through API. +- Registry authentication / write access, or per-registry credentials. +- Multiple registries (keyed `catalog.providers.mcpRegistry.` map). `baseName` is reserved as the per-source mapping-prefix override for a future change. +- Runtime invocation, health checking, or tool discovery of ingested servers. +- Cross-registry dedup/merge of the same server. +- Frontend / catalog UI changes. + +## Decisions + +### D1: Configuration under `catalog.providers.mcpRegistry` (single registry; `baseName` for future multi-registry) + +**Choice:** Config is a single object at `catalog.providers.mcpRegistry` with `baseUrl` (required), `baseName` (optional; passed as the mapping's prefix override), `apiVersion` (default `v1`), `schedule` (`SchedulerServiceTaskScheduleDefinition`), `pageLimit` (optional; max **pages** fetched per sync, default `10`), `pageSize` (optional; sent as the registry `?limit=` page-size query; omitted from the request when unset), and optional `defaultOwner` (entity ref; omit → mapping default `unknown`). A keyed map of instances is rejected at startup. A `config.d.ts` declares the schema so app-config validation and IDE assistance work. + +**Alternatives considered:** (a) The prototype's `mcp.registry.proxy.` namespace — rejected; that namespace reads as proxy/pass-through config and is not where catalog operators look for entity providers. (b) A keyed map `catalog.providers.mcpRegistry.` matching `github`/`ldap` providers — deferred; multiple registries are out of scope for the initial implementation. `baseName` is the hook that change would use so each source can override the mapping `prefix` (`____`). (c) Naming the page cap `limit` — rejected; it collides with the registry `?limit=` query parameter. (d) Always sending a default `?limit=` — rejected; when `pageSize` is unset the registry's own default applies. + +**Rationale:** A flat block matches the initial single-registry scope. `baseName` alongside `baseUrl` is cheap now and avoids a mapping redesign when a later change adds more sources. + +### D2: Package as a `catalog-backend-module` with a single `EntityProvider` + +**Choice:** The plugin is a backend module registered via `createBackendModule` that extends the catalog via `catalogProcessingExtensionPoint.addEntityProvider(...)`, adding one `EntityProvider`. The provider owns `getProviderName()` (`mcp-registry-provider`), which identifies its entity bucket for full-mutation pruning. That same string is also each `DeferredEntity`'s mutation `locationKey` (conflict/claim on the entity ref). Independently, each produced entity carries `backstage.io/managed-by-location` so it is visible in the catalog. + +**Alternatives considered:** (a) One `EntityProvider` per registry id — deferred with multi-registry support; independent schedules and per-source `locationKey` pruning are the reasons to revisit this. (b) A standalone backend plugin with its own router — rejected; ingestion needs the catalog extension point, not an HTTP surface (that would be the proxy pattern). + +**Rationale:** One registry, one provider: scheduling, failure isolation, and a single bucket/`locationKey` stay straightforward. + +### D3: Schedule via `SchedulerService`; omitted `schedule` uses a documented default + +**Choice:** The provider is driven by `scheduler.createScheduledTaskRunner(schedule)` and refreshes on the configured `SchedulerServiceTaskScheduleDefinition`. When `schedule` is omitted, a documented default (e.g. `frequency: { minutes: 30 }`, `timeout: { minutes: 3 }`) is applied rather than failing. The provider connects via the standard `EntityProvider.connect` + scheduled `run()` pattern. + +**Alternative considered:** Require `schedule` and fail if absent — rejected; a sensible default keeps first-run setup simple, consistent with the mapping's "never fail for a supplyable default" stance. + +**Rationale:** A missing schedule should not block ingestion. A sensible default keeps first-run setup simple. + +### D4: Full cursor pagination is mandatory + +**Choice:** The provider loops: request `//servers`, accumulate `servers[]`, read `metadata.nextCursor`, and re-request with `?cursor=` until the cursor is absent, null, or empty. Cursors are opaque and passed verbatim. When `pageSize` is set, every list request in the sync includes `?limit=`. When `pageSize` is omitted, `?limit=` is left unset so the MCP Registry default page size applies. The loop is capped at `pageLimit` pages per sync (`10` when `pageLimit` is omitted). Fetching another page after that cap while `nextCursor` is still set, or detecting a repeated cursor, fails the run (D6) rather than committing a partial catalog. `pageLimit` is never forwarded as `?limit=`. + +**Alternative considered:** Trust a single page (prototype behavior) — rejected; silently truncates ingestion for any registry larger than one page. Unbounded pagination — rejected; a stuck cursor would spin forever. Silently stop at `pageLimit` and commit what was fetched — rejected; that would prune unfetched servers (D6). Invent a default `pageSize` and always send `?limit=` — rejected; the registry's default stands when the operator does not set `pageSize`. + +**Rationale:** Unlike the reference prototype (first page only), cursor pagination continues until the registry is exhausted **or** the page cap is hit. The default of `10` pages is a safe first-run bound; operators raise `pageLimit` for larger registries and set `pageSize` only when they want to override the registry's page size. + +### D5: Delegate wholly to `mcp-registry-server-mapping`; supply `defaultOwner` and `baseName` as caller overrides + +**Choice:** For each server the provider calls the mapping transform, passing `defaultOwner` as the caller-override owner default and, when configured, `baseName` as the caller-override identity prefix (mapping D4). When `baseName` is omitted the mapping's default prefix `mcp.registry` applies. The provider adds only provider-level concerns on top of the transform's output: mutation `locationKey` `mcp-registry-provider` **and** `backstage.io/managed-by-location` on the entity. It never re-derives mapping-owned names, annotations, or `spec.remotes`. + +**Alternative considered:** Inline a copy of the mapping for "performance" — rejected; violates the single-contract goal and would drift. + +**Rationale:** The mapping is the single source of truth for entity shape; a mapping change automatically flows through the provider. `baseName` is how this provider consumes the mapping's prefix override without inventing a second identity scheme. + +### D6: Failure isolation — skip bad entries, fail bad runs atomically + +**Choice:** Two failure tiers: (a) a single server that the mapping rejects (missing required `server.json` field) is logged with an identifying message and skipped; the run proceeds and commits the rest. (b) A registry-level error (unreachable, non-2xx, unparseable body, or pagination-safeguard trip) fails the whole run: **no** `applyMutation` is emitted, so the last-good catalog state is preserved, and the next scheduled tick retries. + +**Alternative considered:** Commit whatever was fetched before an error — rejected; a partial full mutation prunes entities that still exist, causing catalog flapping. + +**Rationale:** Matches the agent-native principle (predictable errors) and the full-mutation model (a partial full mutation would wrongly prune healthy entities). + +### D7: API-version slug is configurable, defaults to `v1`, discrepancy documented + +**Choice:** The endpoint is `//servers` with `apiVersion` defaulting to `v1`. The current reference registry actually serves `/v0` (prototype) or `/v0.1` (docs); the default follows the proposal's stated `v1` and operators override `apiVersion` to match their registry. URL joining normalizes trailing/leading slashes so `baseUrl` with or without a trailing `/` yields exactly one separator. This is captured as a risk below and an open question. + +**Alternative considered:** Default `v0` to match today's registry — considered and deferred to the user's explicit choice of `v1`. + +**Rationale:** Following the proposal's stated `v1` establishes the forward-looking default; operators override `apiVersion` to match their registry's actual version. + +## Risks / Trade-offs + +- **apiVersion default (`v1`) does not match the live registry (`/v0`, `/v0.1`)** → Operators must set `apiVersion` to their registry's actual version; the default is documented and overridable, and startup/first-sync errors name the endpoint that was requested. Revisit the default if the registry standardizes on a version. +- **Non-terminating or repeating cursor from a buggy registry** → D4 `pageLimit` (default `10`) plus repeated-cursor detection trips and fails the run (D6) rather than looping forever or committing a partial catalog. +- **Partial-page fetch failure mid-pagination** → D6 fails the whole run with no mutation, preserving prior catalog state; no partial full mutation is ever committed. +- **`metadata.name` collisions across registries** (same name+version from two registries) → Out of scope: this implementation ingests one registry. `baseName` is the future per-source prefix override so a later multi-registry change can keep identities distinct via mapping D4; true dedup/merge remains deferred. +- **Large registries** → Cursor pagination continues until the cursor ends, but a registry that still has `nextCursor` after `pageLimit` pages (default `10`) fails the sync until the operator raises `pageLimit`. Optional `pageSize` (`?limit=`) and schedule cadence are the other operator levers. Batching/streaming the mutation is a possible future optimization. +- **Mapping contract drift** → The provider depends on `mcp-registry-server-mapping`; because it delegates wholly (D5), a mapping change flows through automatically, but a breaking signature change to the transform would require a coordinated update here. +- **Unauthenticated registry assumption** → Auth is a non-goal; a registry requiring credentials will fail at fetch (D6) until a future auth extension lands. + +## Migration Plan + +Not applicable to existing data — this is additive and introduces no migration of prior state. Deployment: publish the backend module package, add it to the backend via `backend.add(...)`, and configure `catalog.providers.mcpRegistry` with a `baseUrl` (optional `baseName`, `apiVersion`, `schedule`, `pageLimit`, `pageSize`, `defaultOwner`). Rollback: remove the module registration (or the config block); ingested entities are pruned on the next catalog reconciliation because they are provider-managed via `locationKey`. The provider is inert when unconfigured, so shipping the package without config is a safe no-op. + +## Open Questions + +- **apiVersion default** — should the default track the live registry (`v0`/`v0.1`) instead of `v1` once the registry's versioning stabilizes? Currently `v1` per the proposal; revisit when the MCP Registry pins a stable API version. +- **Registry authentication** — token/header auth is out of scope now; what shape (static token, `${ENV}` substitution, Backstage auth integration) should it take when added? +- **Multiple registries** — deferred. A later change would likely restore a keyed `catalog.providers.mcpRegistry.` map, using each instance's `baseName` as the mapping prefix override so `____` stays unique per source. Cross-registry dedup/merge remains a separate question. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/journal.jsonl b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/journal.jsonl new file mode 100644 index 00000000000..0ec58fb5356 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/journal.jsonl @@ -0,0 +1,57 @@ +{"ts":"2026-08-26T03:49:29Z","phase":"proposal","event":"change.created"} +{"ts":"2026-08-26T03:49:30Z","phase":"any","event":"turn.start","input":"Propose mcp-registry-provider: backend provider plugin consuming MCP Registry, mapping server.json to mcp-server API entities, with config baseUrl/schedule/apiVersion/defaultOwner + cursor pagination"} +{"ts":"2026-08-26T03:55:01Z","phase":"any","event":"decision","input":"config placement + apiVersion default","output":"catalog.providers.mcpRegistry. keyed multi-registry map; apiVersion defaults v1 with v0/v0.1 discrepancy documented as risk"} +{"ts":"2026-08-26T03:55:01Z","phase":"any","event":"decision","input":"reference proxy prototype gap","output":"proxy proto never implemented cursor pagination; provider MUST traverse metadata.nextCursor. Provider = catalog entity provider, not proxy"} +{"ts":"2026-08-26T03:58:49Z","phase":"authoring","event":"artifact.added","ref":"proposal.md"} +{"ts":"2026-08-26T03:58:49Z","phase":"authoring","event":"artifact.added","ref":"specs/mcp-registry-provider/spec.md"} +{"ts":"2026-08-26T04:07:07Z","phase":"authoring","event":"artifact.added","ref":"design.md"} +{"ts":"2026-08-26T04:07:07Z","phase":"authoring","event":"artifact.added","ref":"tasks.md"} +{"ts":"2026-08-26T04:07:13Z","phase":"any","event":"turn.end","output":"Created mcp-registry-provider change: proposal, spec, design, tasks. Config catalog.providers.mcpRegistry., cursor pagination, full mutation, delegates to mapping"} +{"ts":"2026-08-26T04:15:15Z","phase":"any","event":"turn.start","input":"/opsx:audit mcp-registry-provider"} +{"ts":"2026-08-26T04:15:15Z","phase":"any","event":"agent.spawned","kind":"audit","count":"1"} +{"ts":"2026-08-26T04:15:15Z","phase":"any","event":"skill.invoked","name":"opsx:audit"} +{"ts":"2026-08-26T04:15:15Z","phase":"any","event":"turn.end","output":"Audit clean (no CRITICAL): 1 WARNING (schedule type name drift), 1 SUGGESTION (cursor safeguard wording). User chose report-only. Wrote audit.md"} +{"ts":"2026-09-14T14:17:24Z","phase":"any","event":"turn.start","input":"Continue mcp-registry-provider: no multi-registry in v1; add baseName alongside baseUrl to override mapping prefix for future multi-registry."} +{"ts":"2026-09-14T14:17:24Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T14:19:59Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/, design.md, tasks.md","input":"No multi-registry in v1; add baseName alongside baseUrl to override mapping prefix","output":"Single-object catalog.providers.mcpRegistry; keyed maps rejected; baseName passed as mapping prefix override"} +{"ts":"2026-09-14T14:20:00Z","phase":"any","event":"decision","input":"Resolve multi-registry scope and how mapping prefix is overridden from provider config","output":"Multi-registry out of scope; flat config; optional baseName overrides mapping prefix; hook for a later change"} +{"ts":"2026-09-14T14:20:00Z","phase":"any","event":"turn.end","output":"Revised provider artifacts: single registry; baseName overrides mapping prefix; keyed maps fail startup"} +{"ts":"2026-09-14T16:17:52Z","phase":"any","event":"turn.start","input":"Audit mcp-registry-provider artifacts for pre-implementation coherence"} +{"ts":"2026-09-14T16:17:53Z","phase":"any","event":"skill.invoked","name":"openspec-audit-change"} +{"ts":"2026-09-14T16:18:27Z","phase":"any","event":"agent.spawned","kind":"audit","count":"1"} +{"ts":"2026-09-14T16:27:05Z","phase":"authoring","event":"artifact.revised","ref":"specs/, proposal.md, .openspec.yaml","input":"Apply pass-1 audit autofix: provider name/locationKey, .server extract, schedule optional, schema","output":"Propagated D2/D3/D5 into spec/proposal; schema rhdh-spec-driven"} +{"ts":"2026-09-14T16:27:27Z","phase":"any","event":"agent.spawned","kind":"audit","count":"1"} +{"ts":"2026-09-14T16:35:32Z","phase":"authoring","event":"artifact.revised","ref":"specs/, tasks.md","input":"Apply pass-2 audit autofix: pagination-safeguard trip and non-2xx in fail-the-run set","output":"Spec D6 error enum includes safeguard trip and non-2xx; task 3.5 matches"} +{"ts":"2026-09-14T16:36:01Z","phase":"any","event":"agent.spawned","kind":"audit","count":"1"} +{"ts":"2026-09-14T16:44:35Z","phase":"authoring","event":"artifact.revised","ref":"design.md, specs/, tasks.md, proposal.md","input":"Pass-3 autofix: cursor stop-set absent/null/empty; spec names D4 loop safeguard","output":"Cursor stop-set unified; spec pagination safeguard is max-pages/total plus repeated-cursor"} +{"ts":"2026-09-14T16:44:35Z","phase":"any","event":"turn.end","output":"Audit clean (no CRITICAL). 4 WARNING, 7 SUGGESTION. Autofix batches applied; remaining findings are judgment calls."} +{"ts":"2026-09-14T17:18:11Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-provider/spec.md","input":"Align baseName metadata.name examples with mapping hash-on-sanitization rule","output":"Provider baseName scenarios expect stem plus hash suffix when name is reverse-DNS"} +{"ts":"2026-09-14T19:55:37Z","phase":"any","event":"turn.start","input":"Continue mcp-registry-provider: task 4.5 must assert locationKey mcp-registry-provider and managed-by-location, matching spec WHEN/THEN"} +{"ts":"2026-09-14T19:55:37Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T19:56:45Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md, specs/mcp-registry-provider/spec.md, design.md","input":"Task 4.5 must assert locationKey mcp-registry-provider and managed-by-location to match spec WHEN/THEN","output":"4.5 asserts both; 4.4/D2/D5/spec use AND not slash; annotation named backstage.io/managed-by-location"} +{"ts":"2026-09-14T19:56:46Z","phase":"any","event":"decision","input":"Are locationKey and managed-by-location interchangeable for provider attribution tests?","output":"No: locationKey claims the entity ref; backstage.io/managed-by-location is catalog-visible; 4.5 asserts both"} +{"ts":"2026-09-14T19:56:46Z","phase":"any","event":"turn.end","output":"Revised 4.5 to assert locationKey and managed-by-location; 4.4 D2 D5 spec aligned; artifacts still complete"} +{"ts":"2026-09-14T20:13:23Z","phase":"any","event":"turn.start","input":"Grill: if a previously mapped server becomes unmappable, does skip + full mutation prune the prior entity?"} +{"ts":"2026-09-14T20:13:23Z","phase":"any","event":"skill.invoked","name":"grilling"} +{"ts":"2026-09-14T20:13:26Z","phase":"any","event":"turn.end","output":"Fact: skip + full mutation already prunes last-good. Asked: prune unmappable prior entity (recommended) or retain last-good?"} +{"ts":"2026-09-14T20:39:35Z","phase":"any","event":"turn.start","input":"Continue mcp-registry-provider: if limit is unset, default to a 10 page limit per sync"} +{"ts":"2026-09-14T20:39:35Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T20:41:16Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/mcp-registry-provider/spec.md, design.md, tasks.md","input":"If limit is not set, default to a 10 page limit per sync","output":"limit is optional max pages per sync, default 10; not registry ?limit=; exceeding fails the run"} +{"ts":"2026-09-14T20:41:16Z","phase":"any","event":"decision","input":"Expose limit as config or keep pagination bounds internal?","output":"Config limit is max pages per sync, default 10 when omitted; not the registry page-size query"} +{"ts":"2026-09-14T20:41:16Z","phase":"any","event":"turn.end","output":"Revised artifacts: omitted limit defaults to 10 pages per sync; trip fails the run; open question closed"} +{"ts":"2026-09-14T20:45:59Z","phase":"any","event":"turn.start","input":"Continue mcp-registry-provider: rename config limit to pageLimit so it is not confused with the registry query param"} +{"ts":"2026-09-14T20:45:59Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T20:47:33Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/mcp-registry-provider/spec.md, design.md, tasks.md","input":"Rename config limit to pageLimit so it is not confused with the registry query param","output":"Config key is pageLimit; registry ?limit= unchanged; D1 rejects naming the page cap limit"} +{"ts":"2026-09-14T20:47:33Z","phase":"any","event":"decision","input":"Should the page-cap config key be limit or pageLimit?","output":"pageLimit; limit collides with the registry page-size query parameter"} +{"ts":"2026-09-14T20:47:33Z","phase":"any","event":"turn.end","output":"Renamed config limit to pageLimit across artifacts; default 10 pages per sync unchanged"} +{"ts":"2026-09-14T20:54:11Z","phase":"any","event":"turn.start","input":"Continue mcp-registry-provider: add pageSize config that sets ?limit=; omit the query when unset"} +{"ts":"2026-09-14T20:54:11Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T20:55:33Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/mcp-registry-provider/spec.md, design.md, tasks.md","input":"Add pageSize under catalog.providers.mcpRegistry to set ?limit=; omit the query when unset","output":"pageSize maps to ?limit=; omitted leaves query unset for registry default; pageLimit stays local"} +{"ts":"2026-09-14T20:55:33Z","phase":"any","event":"decision","input":"How should registry ?limit= page size be configured?","output":"Optional pageSize; when unset omit ?limit= and use the MCP Registry default"} +{"ts":"2026-09-14T20:55:33Z","phase":"any","event":"turn.end","output":"Added optional pageSize as ?limit=; unset omits the query; pageLimit remains the local page cap"} +{"ts":"2026-09-14T21:29:55Z","phase":"any","event":"turn.start","input":"Audit mcp-registry-provider artifacts for pre-implementation coherence"} +{"ts":"2026-09-14T21:29:55Z","phase":"any","event":"skill.invoked","name":"openspec-audit-change"} +{"ts":"2026-09-14T21:39:03Z","phase":"authoring","event":"artifact.revised","ref":"design.md","input":"Apply pass-1 audit autofix: mark defaultOwner optional in D1","output":"D1 now says optional defaultOwner; omit uses mapping default unknown"} +{"ts":"2026-09-14T21:48:35Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-provider/spec.md, tasks.md","input":"Apply pass-2 audit autofix: nextCursor stop-set includes null","output":"Single-page scenario and task 3.6 now stop on absent, null, or empty"} +{"ts":"2026-09-14T22:00:29Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md","input":"Apply pass-3 audit autofix: provider identity, managed-by-location, repeated cursor","output":"Proposal names getProviderName/locationKey, managed-by-location, and repeated-cursor fail"} +{"ts":"2026-09-14T22:07:54Z","phase":"any","event":"turn.end","output":"Audit clean (no CRITICAL). Applied D1 optional defaultOwner, nextCursor null, proposal identity/cursor. Remaining: skip-vs-prune, initialDelay, schedule default, rollback, annotation value."} diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/proposal.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/proposal.md new file mode 100644 index 00000000000..7e570e874c2 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/proposal.md @@ -0,0 +1,57 @@ +# Proposal: MCP Registry Provider + +## Why + +The [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) change defines the pure `server.json` → `mcp-server` `API` entity transform, but explicitly leaves ingestion out of scope: nothing yet fetches entries from an [MCP Registry](https://github.com/modelcontextprotocol/registry) and puts the resulting entities into the Backstage catalog. Without a provider, an operator who points RHDH at a registry gets no catalog entities. This change delivers that missing runtime component — a Backstage catalog **entity provider** that periodically reads a registry's `server.json` entries and populates the catalog with `mcp-server` `API` entities — so MCP servers published to a registry become discoverable in RHDH. + +## What Changes + +- Introduce a **backend catalog entity provider plugin** (a Backstage `catalog-backend-module`) that, on a configured schedule, lists MCP servers from a configured registry and applies the [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) transform to produce `mcp-server` `API` entities, then commits them to the catalog as a **full mutation** (so servers removed from the registry are pruned). The provider's `getProviderName()` and mutation `locationKey` are `mcp-registry-provider`; each entity also carries `backstage.io/managed-by-location`. +- Support **one registry** via idiomatic Backstage entity-provider configuration under `catalog.providers.mcpRegistry` (a single object, not a keyed map), configuring: + - `baseUrl` — the URL of the MCP Registry (**required**). + - `baseName` — optional identity prefix passed through as the [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) caller-override `prefix` (mapping default `mcp.registry` when omitted). Introduced so a future multi-registry change can give each source a distinct `____` without redesigning the transform. + - `schedule` — optional sync frequency as a standard `SchedulerServiceTaskScheduleDefinition` (`frequency`, `timeout`, optional `initialDelay`); when omitted, a documented default is applied rather than failing. + - `apiVersion` — the version segment used in the API endpoint slug; **defaults to `v1`**. + - `pageLimit` — optional max **pages** fetched per sync; **defaults to `10`**. Exceeding it fails the run with no mutation. + - `pageSize` — optional registry page size, sent as `?limit=`; when omitted, `?limit=` is left unset so the MCP Registry default applies. + - `defaultOwner` — the default `spec.owner` (a `User`/`Group` entity reference) applied to every produced `API` entity, passed as the caller-override default into the mapping. +- Implement **cursor pagination**: the servers endpoint (`//servers`) is traversed by passing the prior response's `metadata.nextCursor` as the `cursor` query parameter until the cursor is absent, null, or empty (per the [generic registry API](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/api/generic-registry-api.md#basic-example-list-servers)), or until `pageLimit` (default `10`) would be exceeded — which fails the run. A repeated cursor also fails the run. Optional `pageSize` is sent as `?limit=`; when unset, that query is omitted. +- Specify **resilient, agent-native sync behavior**: a single server entry that fails to map is logged and skipped without aborting the run; a registry transport/protocol error fails that sync run (leaving the prior catalog state intact) and is retried on the next scheduled tick. + +## Capabilities + +### New Capabilities + +- `mcp-registry-provider`: A scheduled Backstage catalog entity provider that reads MCP servers from one configured MCP Registry (with cursor pagination, optional `pageLimit` default `10`, and optional `pageSize` as `?limit=`), maps each `server.json` to an `mcp-server` `API` entity via [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md), and commits them to the catalog as a full mutation — including configuration (`catalog.providers.mcpRegistry` with `baseUrl` / optional `baseName`), scheduling, API-version slug construction, and error handling. + +### Modified Capabilities + +_(none — no long-lived specs exist under `openspec/specs/` yet; this change introduces a new capability and **consumes** the sibling `mcp-registry-server-mapping` capability as its transform.)_ + +## Non-goals + +- **The mapping itself.** The `server.json` → entity transform, annotation projection, secret redaction, and identity/name rules are owned by [`mcp-registry-server-mapping`](../mcp-registry-server-mapping/proposal.md) and consumed here unchanged. +- **A registry proxy or pass-through API.** Unlike the [reference proxy prototype](https://github.com/gabemontero/rhdh-plugins/tree/mcp-reg-proxy-proto), this plugin does not expose registry endpoints through RHDH; it is a one-way ingestion provider into the catalog. +- **Registry authentication / write access.** Assumes an unauthenticated (or externally-fronted) read-only registry endpoint; per-registry auth credentials are a future extension. +- **Runtime invocation, health checking, or tool discovery** of the ingested MCP servers. +- **Multiple registries.** The initial implementation configures and syncs a single registry. A keyed map of instances (`catalog.providers.mcpRegistry.`) is out of scope; `baseName` is the forward-compatible hook so a later change can override the mapping `prefix` per source. +- **Cross-registry deduplication / merge** of the same server published to multiple registries (carried over from the mapping's non-goals; moot while only one registry is ingested). +- **Frontend / catalog UI** changes; produced entities render via existing upstream `mcp-server` `API` entity support. + +## Canonical Touchpoints + +- **PRDs (`specifications/prd/`)**: None +- **ADRs (`specifications/adr/`)**: None +- **Long-lived specs (`openspec/specs/`)**: None (new capability only; `openspec/specs/` does not yet exist) + +**Change type**: feature-spec + +## Impact + +- **Depends on the sibling `mcp-registry-server-mapping` change** for the transform contract; this provider is the first consumer of that mapping and passes `defaultOwner` as the caller-override owner default and, when configured, `baseName` as the caller-override identity prefix. +- **Depends on Backstage backend framework**: the catalog `EntityProvider` interface, `SchedulerService` (`SchedulerServiceTaskScheduleDefinition`), the new backend system (`createBackendModule` / `coreServices`), and `RootConfigService` for reading `catalog.providers.mcpRegistry`. +- **Source API**: MCP Registry generic API — `GET //servers?cursor=` with optional `limit` from `pageSize`; when `pageSize` is unset, `limit` is omitted. Response `{ servers: [...], metadata: { count, nextCursor } }`. Cursors are opaque and traversed until absent. +- **API-version discrepancy** (documented risk): the current reference registry serves `/v0` (proxy prototype) / `/v0.1` (docs), while `apiVersion` defaults to `v1` per this proposal; operators override `apiVersion` to match their registry. +- **Consumers**: RHDH operators who configure a registry; developers and AI agents who then discover MCP servers via catalog search/filter over the `mcp-server` entities and their `modelcontextprotocol.io/*` annotations. +- **Packaging**: a new backend plugin package (Backstage catalog-backend-module naming convention), wired into the backend via `backend.add(...)`. +- **Upstream**: keep aligned with Backstage's entity-provider / scheduler APIs and the MCP Registry generic API as both evolve. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/specs/mcp-registry-provider/spec.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/specs/mcp-registry-provider/spec.md new file mode 100644 index 00000000000..91afe721ba9 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/specs/mcp-registry-provider/spec.md @@ -0,0 +1,184 @@ +## MCP Registry Provider + +This capability defines a Backstage catalog **entity provider** that ingests MCP servers from one configured [MCP Registry](https://github.com/modelcontextprotocol/registry) into the RHDH catalog as `mcp-server` `API` entities. Multiple registries are out of scope for this implementation. + +On a configured schedule, the provider lists the registry's servers (`GET //servers`), traverses all pages via cursor pagination, transforms each `server.json` document into an `mcp-server` `API` entity using the [`mcp-registry-server-mapping`](../../../mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md) contract (supplying the configured `defaultOwner` as the caller-override owner and, when present, `baseName` as the caller-override identity prefix), and commits the full set to the catalog as a single full mutation so that servers removed from the registry are pruned. + +This spec covers configuration, scheduling, registry API interaction (pagination and API-version slug construction), delegation to the mapping transform, catalog mutation semantics, and error handling. It does **not** redefine the `server.json` → entity transform, which is owned by `mcp-registry-server-mapping`. + +--- + +## ADDED Requirements + +### Requirement: Configure a single MCP registry provider + +The provider SHALL read its configuration from `catalog.providers.mcpRegistry` as a **single object** (not a keyed map of instances). The provider SHALL read `baseUrl` (**required**), `baseName` (optional), `apiVersion` (optional, defaulting to the constant `v1`), `schedule` (optional, a standard `SchedulerServiceTaskScheduleDefinition`), `pageLimit` (optional, the max number of pages fetched per sync, defaulting to the constant `10`), `pageSize` (optional, the registry page size sent as the `limit` query parameter; when omitted the provider SHALL leave `?limit=` unset), and `defaultOwner` (optional, a Backstage entity reference). When `baseName` is present, the provider SHALL pass it to `mcp-registry-server-mapping` as the caller-override identity prefix; when it is omitted, the provider SHALL NOT pass a prefix override, so the mapping's default prefix (`mcp.registry`) applies. The provider SHALL register at most one `EntityProvider` whose `getProviderName()` is the constant `mcp-registry-provider`, which SHALL also be the mutation `locationKey` used for full-mutation pruning. When `catalog.providers.mcpRegistry` is absent, the provider SHALL register nothing and SHALL NOT error (the module is inert unless configured). Multiple registries are out of scope: a keyed map of instance objects SHALL fail startup with an actionable error. + +#### Scenario: Single registry configured + +- **WHEN** `catalog.providers.mcpRegistry` is configured with a `baseUrl` and a `schedule` +- **THEN** exactly one provider is registered using the configured `baseUrl` and `schedule`, `apiVersion` defaulting to `v1`, no prefix override beyond the mapping's default, and no owner override beyond the mapping's own default + +#### Scenario: baseName is accepted alongside baseUrl + +- **WHEN** `catalog.providers.mcpRegistry` is configured with `baseUrl` and `baseName: com.example.registry` +- **THEN** the provider is registered, and on sync it passes `com.example.registry` as the mapping's prefix override + +#### Scenario: Keyed multi-registry config is rejected + +- **WHEN** `catalog.providers.mcpRegistry` is a keyed map of instance objects (e.g. keys `internal` and `public`, each with its own `baseUrl`) +- **THEN** provider startup fails with an actionable error that multiple registries are out of scope + +#### Scenario: Missing required baseUrl fails fast + +- **WHEN** `catalog.providers.mcpRegistry` is present but omits `baseUrl` +- **THEN** provider startup fails with an actionable error that names the missing `baseUrl` key + +#### Scenario: No configuration present + +- **WHEN** `catalog.providers.mcpRegistry` is not present in app-config +- **THEN** the module registers no provider and startup succeeds without error + +#### Scenario: Omitted pageLimit defaults to 10 pages per sync + +- **WHEN** `catalog.providers.mcpRegistry` is configured with a `baseUrl` and no `pageLimit` +- **THEN** the provider's max pages per sync is `10` + +#### Scenario: Configured pageLimit is the max pages per sync + +- **WHEN** `catalog.providers.mcpRegistry` is configured with `pageLimit: 3` +- **THEN** the provider's max pages per sync is `3` + +#### Scenario: Omitted pageSize leaves the registry page-size query unset + +- **WHEN** `catalog.providers.mcpRegistry` is configured with a `baseUrl` and no `pageSize` +- **THEN** list requests omit the `limit` query parameter and the MCP Registry default page size applies + +#### Scenario: Configured pageSize is sent as the limit query parameter + +- **WHEN** `catalog.providers.mcpRegistry` is configured with `pageSize: 50` +- **THEN** each servers list request includes `limit=50` + +### Requirement: Sync on the configured schedule + +The provider SHALL run its ingestion sync on the configured `schedule` using the Backstage `SchedulerService`. When `schedule` is omitted, the provider SHALL apply a documented default `SchedulerServiceTaskScheduleDefinition` rather than failing. The provider SHALL also perform an initial sync according to the schedule's `initialDelay` (or immediately when unset) after registration. + +#### Scenario: Scheduled sync runs at the configured frequency + +- **WHEN** the provider is configured with a `schedule` of `frequency: { minutes: 30 }` +- **THEN** the provider runs a full ingestion sync approximately every 30 minutes via the scheduler + +#### Scenario: Schedule omitted uses the default + +- **WHEN** the provider is configured without a `schedule` +- **THEN** the provider applies the documented default schedule and syncs on that cadence without error + +### Requirement: List registry servers with cursor pagination + +During a sync, the provider SHALL request the registry's servers from `//servers` and SHALL traverse pages using the registry's cursor pagination: it SHALL read `metadata.nextCursor` from each response and, when that value is present and non-empty, issue the next request with that value as the `cursor` query parameter, repeating until `metadata.nextCursor` is absent, null, or empty. Cursors SHALL be treated as opaque strings (never constructed or modified). All `servers[]` entries across fetched pages SHALL be accumulated for the sync. When `pageSize` is set, the provider SHALL send it as the `limit` query parameter on every list request in the sync. When `pageSize` is omitted, the provider SHALL omit `?limit=` so the MCP Registry default page size applies. The provider SHALL NOT send `pageLimit` as the registry `?limit=` query parameter; `pageLimit` is a local max-pages bound only. The provider SHALL fail the run with no mutation when it would fetch more pages than `pageLimit` (default `10`) while `metadata.nextCursor` is still present and non-empty, or when a repeated cursor is detected. + +#### Scenario: Multi-page traversal + +- **WHEN** the registry returns a first page with `metadata.nextCursor` set and a second page with no `nextCursor` +- **THEN** the provider fetches both pages, passing the first page's `nextCursor` as the `cursor` parameter on the second request, and accumulates the servers from both pages + +#### Scenario: Single-page result + +- **WHEN** the registry returns a page whose `metadata.nextCursor` is absent, null, or empty +- **THEN** the provider stops after that single request and processes only the accumulated servers + +#### Scenario: Opaque cursor is passed unchanged + +- **WHEN** a response's `metadata.nextCursor` is an opaque token +- **THEN** the provider passes that token verbatim as the `cursor` query parameter without parsing or altering it + +#### Scenario: Omitted pageSize does not send limit on list requests + +- **WHEN** `pageSize` is omitted +- **THEN** each servers list request has no `limit` query parameter + +#### Scenario: Configured pageSize is present on every page request + +- **WHEN** the provider is configured with `pageSize: 50` and the registry returns a first page with `metadata.nextCursor` set +- **THEN** both the first request and the follow-up request include `limit=50` + +#### Scenario: Default pageLimit trips when an 11th page is required + +- **WHEN** `pageLimit` is omitted (default `10`) and the 10th page still has a non-empty `metadata.nextCursor` +- **THEN** the provider does not fetch an 11th page, logs the pagination-safeguard trip, and does not commit a mutation for this run + +#### Scenario: Configured pageLimit trips before the cursor ends + +- **WHEN** the provider is configured with `pageLimit: 2` and the 2nd page still has a non-empty `metadata.nextCursor` +- **THEN** the provider does not fetch a 3rd page, logs the pagination-safeguard trip, and does not commit a mutation for this run + +### Requirement: Construct the servers endpoint from apiVersion + +The provider SHALL construct the servers endpoint as `//servers`, where `apiVersion` is the configured value or the `v1` default. The provider SHALL join `baseUrl` and the version segment without duplicating or dropping path separators, regardless of whether `baseUrl` has a trailing slash. + +#### Scenario: Default apiVersion + +- **WHEN** the provider is configured with `baseUrl: https://registry.example.com` and no `apiVersion` +- **THEN** the provider requests `https://registry.example.com/v1/servers` + +#### Scenario: Overridden apiVersion + +- **WHEN** the provider is configured with `baseUrl: https://registry.example.com/` (trailing slash) and `apiVersion: v0` +- **THEN** the provider requests `https://registry.example.com/v0/servers` with exactly one separator between segments + +### Requirement: Map each registry server to an mcp-server API entity + +For every accumulated server entry, the provider SHALL extract the `server.json` document from the list entry's `.server` object and produce an `mcp-server` `API` entity by applying the `mcp-registry-server-mapping` transform, supplying the configured `defaultOwner` as the caller-override owner default and, when `baseName` is present, supplying `baseName` as the caller-override identity prefix. The provider SHALL NOT reimplement or alter the field mapping, annotation projection, or identity rules defined by `mcp-registry-server-mapping`. Each produced entity SHALL be emitted with mutation `locationKey` `mcp-registry-provider` **and** SHALL carry `backstage.io/managed-by-location` so the catalog attributes the entity to this provider. + +#### Scenario: Server mapped with configured default owner + +- **WHEN** a sync retrieves a `server.json` and the provider is configured with `defaultOwner: group:default/mcp-admins` +- **THEN** the produced `mcp-server` `API` entity has `spec.owner: group:default/mcp-admins` (the caller override), with all other fields set by the `mcp-registry-server-mapping` transform + +#### Scenario: Default owner omitted falls back to the mapping default + +- **WHEN** a sync retrieves a `server.json` and the provider is configured with no `defaultOwner` +- **THEN** the produced entity's `spec.owner` is the `mcp-registry-server-mapping` default (`unknown`) + +#### Scenario: baseName overrides the mapping identity prefix + +- **WHEN** a sync retrieves a `server.json` with `name: io.github.user/weather` and `version: 1.0.2`, and the provider is configured with `baseName: com.example.registry` +- **THEN** the mapping is invoked with prefix override `com.example.registry` and the produced entity's `metadata.name` is the sanitized stem `com.example.registry__io.github.user-weather__1.0.2` plus a stable hash suffix (sanitization changed the identity) + +#### Scenario: baseName omitted uses the mapping default prefix + +- **WHEN** a sync retrieves a `server.json` with `name: io.github.user/weather` and `version: 1.0.2`, and the provider is configured with no `baseName` +- **THEN** the mapping is invoked with no prefix override and the produced entity's `metadata.name` uses the mapping default prefix stem (`mcp.registry__io.github.user-weather__1.0.2`) plus a stable hash suffix (sanitization changed the identity) + +#### Scenario: Provider attribution annotations present + +- **WHEN** the provider produces an entity +- **THEN** the entity's mutation `locationKey` is `mcp-registry-provider` and the entity carries `backstage.io/managed-by-location` so the catalog associates the entity with this provider and can prune it on removal + +### Requirement: Commit ingested entities as a full mutation + +At the end of each successful sync, the provider SHALL commit the complete set of produced entities to the catalog as a single **full** mutation (not incremental), so that entities for servers no longer present in the registry are removed from the catalog and re-added/updated entities reflect the latest `server.json`. The provider SHALL NOT emit a mutation for a sync run that failed to complete (see error handling), leaving the prior catalog state intact. + +#### Scenario: Removed server is pruned + +- **WHEN** a server present in a prior sync is absent from the current sync's accumulated servers +- **THEN** the current sync's full mutation omits that server's entity, and the catalog removes it + +#### Scenario: Updated server reflects latest state + +- **WHEN** a server's `server.json` changes between syncs (e.g. a new description) +- **THEN** the produced entity in the current full mutation reflects the latest `server.json` + +### Requirement: Resilient, agent-native error handling + +A single server entry that cannot be mapped (e.g. it omits a `server.json`-required field and the mapping rejects it) SHALL be logged with an actionable message identifying the entry and SHALL be skipped, without aborting the sync or discarding the other entries. A registry transport or protocol error (unreachable host, non-2xx HTTP status, unparseable response body, or pagination-safeguard trip) SHALL fail the current sync run: the provider SHALL NOT commit a partial full mutation, SHALL log the error, and SHALL retry on the next scheduled tick, leaving the prior catalog state intact. + +#### Scenario: One malformed server does not abort the sync + +- **WHEN** one accumulated server entry fails mapping while the others succeed +- **THEN** the provider logs the failing entry, skips it, and commits a full mutation containing the successfully mapped entities + +#### Scenario: Registry fetch error aborts the run without a mutation + +- **WHEN** a page request returns a non-2xx HTTP status or the response body cannot be parsed +- **THEN** the provider logs the error, does not commit any mutation for this run, leaves the previously committed catalog entities intact, and retries on the next scheduled tick diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/tasks.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/tasks.md new file mode 100644 index 00000000000..d427af57000 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-provider/tasks.md @@ -0,0 +1,51 @@ +# Tasks: MCP Registry Provider + + + + +## 1. Plugin Scaffolding & Packaging + +- [ ] 1.1 Create the backend `catalog-backend-module` plugin package (Backstage catalog-backend-module naming convention) with `package.json`, `tsconfig`, and lint config matching the workspace's plugin conventions +- [ ] 1.2 Add the `createBackendModule` skeleton that registers against `catalogProcessingExtensionPoint`, depending on `coreServices` (`rootConfig`, `logger`, `scheduler`) +- [ ] 1.3 Document installation in the plugin `README.md` (add via `backend.add(...)`, minimal app-config example) + +## 2. Configuration + +- [ ] 2.1 Author `config.d.ts` declaring `catalog.providers.mcpRegistry` as a single object with `baseUrl` (required), `baseName?` (optional mapping-prefix override), `apiVersion?` (default `v1`), `schedule?` (`SchedulerServiceTaskScheduleDefinitionConfig`), `pageLimit?` (max pages per sync, default `10`), `pageSize?` (registry `?limit=` when set), and `defaultOwner?`; require `@visibility backend` annotations for backend-only fields such as `baseUrl` +- [ ] 2.2 Implement config reading: parse `catalog.providers.mcpRegistry` as a single object; register nothing (no error) when the key is absent +- [ ] 2.3 Implement validation with actionable errors (fail fast when `baseUrl` is missing; fail fast with a multiple-registries-out-of-scope message when the value is a keyed map of instance objects); apply the `apiVersion` default (`v1`), the documented default schedule, and the `pageLimit` default (`10` pages per sync) when omitted +- [ ] 2.4 Add unit tests for config parsing/validation: single object with `baseUrl`, optional `baseName`, keyed-map rejection, missing `baseUrl`, absent-config no-op, omitted `pageLimit` → `10`, explicit `pageLimit` override, omitted `pageSize` (no invented default), and explicit `pageSize` + +## 3. Registry Client & Pagination + +- [ ] 3.1 Define the registry API response types (`servers[]`, `metadata.count`, `metadata.nextCursor`) and the `server.json` extraction from each `servers[]` entry (`.server`) +- [ ] 3.2 Implement servers-endpoint URL construction `//servers` with slash normalization (works with and without a trailing slash on `baseUrl`) +- [ ] 3.3 Implement cursor pagination: loop passing prior `metadata.nextCursor` as the `cursor` query param until it is absent, null, or empty, accumulating all `servers[]`; treat cursors as opaque; when `pageSize` is set send it as `?limit=` on every list request; when `pageSize` is omitted leave `?limit=` unset +- [ ] 3.4 Implement the pagination loop safeguard: cap fetches at configured `pageLimit` pages per sync (`10` when omitted); do not send `pageLimit` as the registry `?limit=` query param; detect a repeated cursor; exceeding the page cap or a repeated cursor fails the run rather than looping forever +- [ ] 3.5 Implement registry-error handling (unreachable host, non-2xx status, unparseable body, pagination-safeguard trip) raising a typed error that aborts the run +- [ ] 3.6 Add unit tests for the client using mocked HTTP: single page, multi-page traversal, empty/absent/null cursor termination, opaque-cursor passthrough, omitted `pageSize` (no `limit` query), configured `pageSize` as `limit` on every page request, default `pageLimit` `10` tripping on an 11th page, configured `pageLimit` tripping, and error/safeguard cases + +## 4. Entity Provider & Scheduling + +- [ ] 4.1 Implement the `EntityProvider` class: `getProviderName()` = `mcp-registry-provider`, `connect()` storing the connection, and a `run()` performing one sync +- [ ] 4.2 Wire scheduling via `SchedulerService.createScheduledTaskRunner(schedule)`, honoring `initialDelay`; register the single provider when config is present +- [ ] 4.3 Implement the full-mutation commit: on successful sync call `connection.applyMutation({ type: 'full', entities })`; on a failed run emit no mutation (preserve prior catalog state) +- [ ] 4.4 Attach provider attribution to each entity: set the mutation `locationKey` to `mcp-registry-provider` **and** set `backstage.io/managed-by-location` on the entity (they are not interchangeable — the key claims the entity ref; the annotation is catalog-visible source) +- [ ] 4.5 Add unit tests: full mutation contents (each `DeferredEntity` has `locationKey` `mcp-registry-provider` **and** the entity carries `backstage.io/managed-by-location`, matching scenario `Provider attribution annotations present`), pruning of removed servers across two syncs, updated server reflected, and no-mutation-on-failed-run + +## 5. Mapping Integration + +- [ ] 5.1 Depend on the sibling `mcp-registry-server-mapping` transform and invoke it per accumulated server, passing `defaultOwner` as the caller-override owner default and, when configured, `baseName` as the caller-override identity prefix (never reimplement the mapping) +- [ ] 5.2 Implement per-entry failure isolation: catch a mapping rejection (e.g. missing required `server.json` field), log an actionable message identifying the entry, skip it, and continue the run +- [ ] 5.3 Add integration tests over sample `server.json` inputs → produced `mcp-server` `API` entities, asserting `spec.owner` reflects `defaultOwner` (and the mapping default `unknown` when omitted), `metadata.name` uses `baseName` as prefix when configured (and mapping default `mcp.registry` when omitted), and that one bad entry does not abort the batch + +## 6. End-to-End Verification & Docs + +- [ ] 6.1 Add an end-to-end test wiring config → mocked paginated registry → mapping → full mutation, asserting the mutation converges to the registry's current server set +- [ ] 6.2 Verify produced entities pass the upstream `mcp-server` `API` entity schema (`McpServerApiEntity`) — reusing the mapping change's conformance expectations +- [ ] 6.3 Verify the apiVersion discrepancy handling: default `v1` requests `/v1/servers` and an override (`v0`) is honored, with a documented note for operators +- [ ] 6.4 Finalize `README.md` / config docs: full `catalog.providers.mcpRegistry` example (`baseUrl`, optional `baseName`, `apiVersion`, `schedule`, `pageLimit` default `10` pages per sync, optional `pageSize` as `?limit=`, `defaultOwner`), note that multiple registries are out of scope, pagination behavior, and error-handling semantics +- [ ] 6.5 Run the workspace lint, typecheck, and test suite; ensure the new package builds and passes CI conventions diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/.openspec.yaml b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/.openspec.yaml new file mode 100644 index 00000000000..adc5812c689 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/.openspec.yaml @@ -0,0 +1,3 @@ +schema: spec-driven +created: 2026-08-18 +status: draft diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/audit.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/audit.md new file mode 100644 index 00000000000..a6a91c5ce85 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/audit.md @@ -0,0 +1,32 @@ +## Audit Report: mcp-registry-server-mapping + +**Last audited:** 2026-09-14T21:28:22Z + +### Summary + +| Category | CRITICAL | WARNING | SUGGESTION | +| -------- | -------- | ------- | ---------- | +| A | 0 | 0 | 0 | +| B | 0 | 0 | 0 | +| C | 0 | 3 | 0 | +| D | 0 | 0 | 1 | +| E | 0 | 0 | 0 | +| F | 0 | 1 | 0 | +| G | 0 | 0 | 1 | +| H | 0 | 0 | 0 | + +### CRITICAL + +- None + +### WARNING + +- **C** `openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-annotation-projection/spec.md:80` — `the projection SHALL be skipped or disambiguated`. Pick one reserved-key collision behavior. Skip only when re-projecting a field already consumed by the direct mapping (matches Direct-mapping annotation wins). Require D3 hash-suffix disambiguation when a different source path sanitizes to a reserved key; skip in that case would violate Scalar round-trip fidelity. +- **C** `openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-annotation-projection/spec.md:113` — `Null values and empty containers MAY be omitted per a documented rule`. Replace MAY with SHALL to match the Nulls and empty containers scenario, and state the rule in this requirement (what is omitted vs retained: null, [], {} vs empty string, 0, false). Restate that rule in design.md and tasks 1.4/3.3; they currently cite a rule that is not written down. +- **C** `openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md:68` — `stem mcp.registry__io.github.user-weather__1.0.2 plus a stable hash suffix`. State in the identity requirement and design D4 the substitutions that produce this stem (at least `/` → `-`). “Sanitized” plus the Backstage charset does not uniquely determine that THEN; other replacements would still be catalog-valid. +- **F** `openspec/changes/mcp-registry-server-mapping/.openspec.yaml:1` — `schema: spec-driven`. Set `schema` to `rhdh-spec-driven` to match `openspec/config.yaml` and `openspec/schemas/rhdh-spec-driven/schema.yaml`. + +### SUGGESTION + +- **G** `openspec/changes/mcp-registry-server-mapping/design.md:63` — `Path segments are sanitized (illegal characters and leading _ replaced)`. Specify the replacement character(s) for illegal annotation-key characters and leading `_`, then copy the same substitutions into the annotation-projection Produce catalog-valid annotation keys requirement and task 3.2. The `_meta` scenario only says “allowed characters.” +- **D** `openspec/changes/mcp-registry-server-mapping/proposal.md:1` — `# Proposal: MCP Registry Server Mapping`. Shorten proposal.md to the config.yaml proposal rule of under 500 words. Move mapping-table detail and long upstream schema excerpts into design.md or the mapping spec. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/design.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/design.md new file mode 100644 index 00000000000..fa3dfac824d --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/design.md @@ -0,0 +1,157 @@ +# Design: MCP Registry Server Mapping + +## Canonical Touchpoints + +Carried forward from the proposal: + +- **PRDs (`specifications/prd/`)**: None +- **ADRs (`specifications/adr/`)**: None +- **Long-lived specs (`openspec/specs/`)**: None + +No canonical document updates. This change introduces new capabilities only and does not modify any existing canonical document or long-lived spec. + +## Context + +MCP registries publish server entries as `server.json` documents conforming to the draft [`server.schema.json`](https://raw.githubusercontent.com/modelcontextprotocol/registry/refs/heads/main/docs/reference/server-json/draft/server.schema.json). Backstage already supports cataloging MCP servers as `API` entities with `spec.type: mcp-server` (implemented upstream in `backstage/backstage`, not by RHDH). This change defines the pure transform between the two — one `server.json` in, one `API` entity out — and nothing else. Who fetches registry entries and when (an entity provider, a polling schedule) is explicitly out of scope. + +Upstream Backstage defines a **dedicated `mcp-server` `API` entity schema** that overrides the base `API` schema for `spec.type: mcp-server`: + +- The **dedicated mcp-server entity schema** [`McpServerApiEntity.ts`](https://github.com/backstage/backstage/blob/f91434377dc43cd64bef82344e3f2b539bfdaf11/packages/catalog-model/src/kinds/McpServerApiEntity.ts#L28-L36) (Backstage [PR #34016](https://github.com/backstage/backstage/pull/34016)) does `Omit` and redefines `spec` for `spec.type: mcp-server` as required `type`, `lifecycle`, `owner`, and `remotes[]` (with optional `system`). `spec.remotes` is **required** and `spec.definition` is **not part of this schema** — the dedicated schema replaces (not merely extends) the base `spec`. +- The **canonical example** [`backstage-mcp-server-api.yaml`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/examples/apis/backstage-mcp-server-api.yaml) matches that schema: top-level `spec.remotes[]` (`type`, `url`) and no `spec.definition`. +- The committed **base API schema** [`API.v1alpha1.schema.json`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json) lists `definition` among the required `spec` fields and does not define `spec.remotes` — but that `definition` requirement does **not** apply to `spec.type: mcp-server`, which is governed by the dedicated schema above. + +The mapping therefore targets the dedicated `mcp-server` entity shape: top-level `spec.remotes[]` in place of `spec.definition`. This aligns with the upstream-first principle — the shape is the one upstream ships, not an RHDH invention. + +## Goals / Non-Goals + +**Goals:** + +- A deterministic, idempotent, side-effect-free transform: `server.json` (+ caller defaults) → one `mcp-server` `API` entity. +- Faithful adherence to the upstream mcp-server entity shape (top-level `spec.remotes[]`, no `spec.definition`). +- Collision-free entity identity across multiple versions of the same server (`____`, prefix default `mcp.registry`). +- Lossless capture of source data: every non-null scalar leaf is recoverable from a native field or a `modelcontextprotocol.io/*` annotation, except D9-redacted `isSecret` `default`/`value`/`choices` leaves and D11-refused URL scalars (not copied into emitted URL fields and not projected as annotations). +- Catalog-valid output: every produced key/name passes Backstage validation. + +**Non-Goals:** + +- Registry HTTP client, polling, scheduling, or an entity provider/processor (separate future change). +- Modifying the upstream `mcp-server` entity contract or its validation. +- Reverse mapping (entity → `server.json`) beyond the scalar round-trip guarantee. +- Executing or health-checking mapped servers, or interpreting local `packages[]` runtime details. +- Deduplicating the same server across multiple registries. + +## Decisions + +### D1: Target the upstream example shape — top-level `spec.remotes[]`, no `spec.definition` + +**Choice:** The entity carries `spec.type: mcp-server`, `spec.lifecycle`, `spec.owner`, and top-level `spec.remotes[]` (`type`, `url`). No `spec.definition` is emitted. + +**Alternatives considered:** (a) Inline `spec.definition` MCP Server Specification string — rejected; the mcp-server entity uses `spec.remotes` instead. (b) Dual-write both `spec.definition` and `spec.remotes` for base-schema safety — rejected as redundant given the target shape uses `spec.remotes`. + +**Rationale:** Aligns with the dedicated upstream `McpServerApiEntity` schema, which defines `spec.remotes` in place of `spec.definition`. + +### D2: Two-tier mapping — native lift, then annotation projection fallback + +**Choice:** Attributes with a native home lift into `metadata`/`spec` fields or dedicated annotations (`mcp-registry-server-mapping`); everything else is projected into `modelcontextprotocol.io/*` annotations (`mcp-registry-annotation-projection`). Dedicated annotations include `modelcontextprotocol.io/name`, `modelcontextprotocol.io/version`, and the unnormalized `modelcontextprotocol.io/repository.url` when that URL passes D11. A URL that D11 refuses is not copied into any emitted URL field and is not projected as an annotation. + +**Alternative considered:** Stuff the entire `server.json` into a single annotation blob — rejected; opaque and not individually searchable/filterable. + +**Rationale:** Keeps the entity idiomatic for catalog consumers while losing no source data. + +### D3: Annotation key encoding — dot-separated path after the prefix + +**Choice:** Nested paths are encoded as `modelcontextprotocol.io/attribute.tree.to.leaf` (object keys by name, array elements by zero-based index). Path segments are sanitized (illegal characters and leading `_` replaced) and over-length keys are truncated with a stable hash suffix; sanitization collisions are disambiguated by the same hash suffix. + +**Alternatives considered:** Literal slashes (`.../attr/tree/leaf`) — rejected, invalid Backstage keys; hyphenated scalars + JSON blobs for arrays — rejected, less uniform and less queryable. + +**Rationale:** Backstage annotation keys allow exactly one `/` and a ≤63-char name segment over a restricted character set, making dot-separated encoding the most uniform and queryable representation that fits within those constraints. + +### D4: Entity identity — `metadata.name` = `____` + +**Choice:** `metadata.name` is the sanitized prefix, sanitized canonical name, and sanitized version joined by `__`. The prefix is the constant `mcp.registry` by default; a caller MAY supply an override default (same caller-override pattern as owner and lifecycle in D5). If the override is unset, empty, or sanitizes to empty, the mapping uses `mcp.registry` — it never fails for a missing prefix, and the produced name never starts with `_`. The bare canonical name is preserved in `modelcontextprotocol.io/name` and the version in `modelcontextprotocol.io/version`, so both remain individually queryable and the identity is reconstructable together with the effective prefix. A stable hash suffix derived from the prefix, canonical name, and version is appended whenever sanitization mutates any identity segment or the joined candidate exceeds 63 characters (truncate the stem as needed). An already catalog-valid candidate that is ≤63 characters is emitted with no hash. The rule is per-input; it does not observe other documents. + +**Alternatives considered:** (a) Encode the version in `metadata.namespace` — rejected; fragments entity references and complicates relationships. (b) `__` with no prefix — rejected; leaves registry-mapped entities without a caller-controllable namespacing token in `metadata.name` (they would collide with any other `mcp-server` API that sanitizes to the same name+version). + +**Rationale:** A registry publishes one `server.json` per version and each becomes its own entity, so a name derived from the canonical name alone would collide across versions. The prefix distinguishes registry-mapped entities in a shared catalog and lets the future ingestion layer pass a per-source override without changing the transform. + +### D5: Supplying fields absent from `server.json` — owner and lifecycle + +**Choice:** `spec.owner` is set to the constant `unknown` by default; a caller MAY supply an override default, but the transform never fails for a missing owner (a placeholder owner keeps the output valid, and the future ingestion change can reassign ownership). `spec.lifecycle` is set to the constant `production` by default; a caller MAY supply an override default lifecycle value. Both fields use the same caller-override pattern as the identity prefix in D4. + +**Alternatives considered:** (a) Require caller-provided owner/lifecycle and fail if absent — rejected; a pure transform should always yield a valid entity, and ownership/lifecycle assignment belongs to the ingestion layer. (b) Derive lifecycle from a `status` field — rejected; `status` is not part of the base `server.schema.json` (verified 2026-08-21 against the draft schema). + +**Rationale:** `server.json` (per the base `server.schema.json`) has no owner or lifecycle fields. Defaults keep the output valid without requiring the caller to supply values that the ingestion layer should own. + +### D6: Determinism and idempotency + +**Choice:** The transform is a pure function of (`server.json`, caller defaults) with stable ordering of `spec.remotes`, `metadata.tags`, and annotation keys, and no timestamps or randomness. + +**Rationale:** Makes the output safe as the identity for repeated ingestion and usable as a golden-file test oracle. + +### D7: Fail-open to generic projection on schema drift + +**Choice:** Native mappings are pinned to known fields; any field not recognized by a native rule is still captured by the generic annotation projection **unless** D11 refuses it as a URL. New/unknown source fields are therefore never dropped except D11-refused URL scalars — at worst they land in annotations rather than a native field. + +**Rationale:** The draft `server.schema.json` evolves. Fail-open projection ensures no source data is lost as the schema changes, except URL scalars D11 refuses. + +### D8: Servers with no remotes emit an empty `spec.remotes: []` + +**Choice:** A `server.json` that declares no `remotes` (e.g. a local-`packages`-only server), or whose remotes are all refused by the emitted-URL scheme policy (D11), is mapped to an entity with an **empty array** `spec.remotes: []`, never an omitted field. + +**Alternative considered:** Treat a no-remotes server as a mapping failure — rejected; such servers are valid registry entries and their `packages`/metadata are still worth cataloging (their runtime details are preserved via annotation projection). + +**Rationale:** The upstream `McpServerApiEntity` schema requires `spec.remotes`. An empty array keeps the output both schema-conformant and deterministic (D6) — the no-remotes case has a single, stable representation. + +**Dependency note:** This assumes `McpServerApiEntity` accepts an empty `remotes` array (no `minItems: 1`); if upstream later enforces a non-empty `remotes`, revisit this decision (emit a failure or a documented placeholder). + +### D9: Redact secret-flagged input values from annotation projection + +**Choice:** When an input object declares `isSecret: true`, the projection walker SHALL prune (omit) that object's `default`, `value`, and `choices` leaves (including every `choices[]` element); those values SHALL NOT appear in any `modelcontextprotocol.io/*` annotation. Non-secret sibling leaves (`name`, `description`, `format`, `isRequired`, `isSecret` itself, `placeholder`, …) continue to project. The redaction applies uniformly to every `isSecret`-bearing input, not only environment variables — redacting env vars while leaving remote `headers`/`variables` exposed would reintroduce the same leak. + +**Alternatives considered:** (a) Hash/mask the value instead of omitting — rejected; a mask still advertises the secret's presence and length without adding catalog value, and a hash is neither reversible nor useful for discovery. (b) Project into a differently-prefixed "sensitive" annotation — rejected; catalog annotations are not a secret store, so any in-entity representation is unsafe. (c) Keep projecting `choices` on `isSecret: true` inputs and document that they cannot be secrets — rejected; the MCP `Input.choices` field is "a list of possible values for the input," the same class of data as `default`/`value`. Enumerating those values in the catalog would publish the secret's allowed set. + +**Rationale:** The `server.json` `Input` shape (used by `packages[].environmentVariables[]`, `remotes[].headers[]`, `remotes[].variables`, and package/runtime arguments) carries an `isSecret` flag alongside `default`/`value`/`choices`. Because annotation projection (D2/D3) emits scalar leaves into **searchable, plaintext** catalog annotations, projecting those leaves of an `isSecret: true` input would publish a credential (or its enumerated possible values). `placeholder` is guidance about expected form, not a value list, and still projects. + +**Round-trip consequence:** This is a deliberate exception to the scalar round-trip fidelity guarantee in the projection spec; pruned `default`/`value`/`choices` leaves are intentionally _not_ recoverable from the entity. + +### D10: Repository emits both `backstage.io/source-location` and a titled `metadata.links` entry; `websiteUrl` link titled "Website" + +**Choice:** `repository.url` combined with `repository.subfolder` when present is emitted **both** as the canonical Backstage `backstage.io/source-location` annotation — whose value MUST use the `url:` format (`url:`), the format source-aware Backstage tooling (source view, scaffolder, TechDocs) reads to locate an entity's repository — **and** as a human-visible `metadata.links` entry titled `Source Code`, **only when** the original `repository.url` passes D11. Independently, when that URL passes D11, the original `repository.url` scalar is copied **verbatim** (no trailing-`/` or `.git` strip, no subfolder join) into a dedicated `modelcontextprotocol.io/repository.url` annotation so the source value remains recoverable after combination-base normalization. When D11 refuses `repository.url`, the mapping omits the Source Code link, `backstage.io/source-location`, **and** `modelcontextprotocol.io/repository.url`, and does not project the refused URL. `websiteUrl` is emitted as a `metadata.links` entry titled `Website` only when it passes D11; a refused `websiteUrl` is not projected. + +The combined URL is computed by the SCM-aware algorithm in `mcp-registry-server-mapping`: strip trailing `/` and a trailing `.git` from a **copy** of `repository.url` used only as the combination `base`; if `subfolder` is absent the result is that base URL; otherwise select a browse-path template from `repository.source` (`github` → `{base}/tree/HEAD/{subfolder}`, `gitlab` → `{base}/-/tree/HEAD/{subfolder}`, `bitbucket` → `{base}/src/HEAD/{subfolder}`, `azure-devops` → `{base}?path=/{subfolder}`) and fall back to `{base}/{subfolder}` for any other source. The mapping never invents a branch name such as `main` — `server.json` has no ref field, so templates that need a ref use the git symbolic ref `HEAD`. Normalization of the combination `base` SHALL NOT rewrite `modelcontextprotocol.io/repository.url`. + +**Alternatives considered:** (a) Emit only the `metadata.links` source entry and omit `backstage.io/source-location` — rejected; without the canonical annotation, upstream source-location tooling cannot resolve the repository. (b) Emit only the annotation and no link — rejected; the annotation is not surfaced as a browsable link in the catalog UI. (c) Expect `repository.url` to already include a tree/branch path when a subfolder is relevant — rejected; the MCP schema defines `url` as the repository root and `subfolder` as a separate relative path (see the [monorepo example](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/server-json/generic-server-json.md)). (d) Always path-join `url` and `subfolder` with no SCM template — rejected; GitHub/GitLab/Bitbucket/Azure DevOps each use a different subdirectory browse URL, so a naive join is not a valid browse URL on those hosts. (e) Insert a guessed default branch (`main`) into the path — rejected; the default branch is not in `server.json` and is often not `main`. (f) Treat the normalized combined URL as the recoverable `repository.url` and skip a dedicated annotation — rejected; stripping `.git`/`/` (and folding in `subfolder`) would make the original scalar unrecoverable. + +**Rationale:** Emitting the source-location annotation in addition to the link keeps the entity idiomatic for both machine consumers (the annotation) and the catalog UI (the titled link). Keying the join on `repository.source` keeps subdirectory links valid across SCMs without requiring a branch that the schema does not provide. When D11 passes, the original `repository.url` is a dedicated annotation so scalar round-trip still holds after combination-base normalization. `backstage.io/source-location` and `modelcontextprotocol.io/repository.url` (when emitted) are set by the direct mapping and are therefore reserved annotations that the generic projection (D2/D3) must not overwrite or re-derive. D11 gates the dedicated annotation as well as the link and source-location so a `javascript:`/`data:` `repository.url` cannot land in any URL-shaped field. + +### D11: Allowlist `http`/`https` on every emitted or projected URL; do not classify hosts; do not fetch + +**Choice:** Before copying a URL into `metadata.links[].url`, `backstage.io/source-location` (the target after `url:`), `spec.remotes[].url`, `modelcontextprotocol.io/repository.url`, **or any generic `modelcontextprotocol.io/*` annotation whose value is that URL**, the mapping parses the candidate as an absolute URL with no base (trim whitespace; WHATWG `URL` parse). The candidate is copied only when parse succeeds and the scheme is `http` or `https` (case-insensitive). `javascript:`, `data:`, `file:`, `vbscript:`, `blob:`, relative paths, scheme-relative `//host`, and scp-like `git@host:path` are refused. A refusal omits that emitted field (and omits a refused remote from `spec.remotes`) but does **not** fail the transform. + +Generic projection SHALL NOT emit a D11-refused URL under any key. That includes refused `websiteUrl`, refused `remotes[].url`, refused `repository.url`, refused `icons[].src`, and any other URL-typed leaf (known fields such as `websiteUrl` / `*.url` / `*.src`, and any scalar that WHATWG-parses as a URL whose protocol is not `http:` or `https:`). Non-URL siblings still project (`remotes[].type`, `icons[].mimeType`, `repository.source` / `id` / `subfolder`). D11-refused URL scalars are a deliberate round-trip exception. Plain non-URL strings (package identifiers, descriptions) are unchanged: a failed absolute-URL parse is not a reason to drop them. + +The scheme gate does **not** classify hosts as public vs private and does not treat “internal-looking” as a special case. Fixtures such as `http://localhost:7007/api/mcp/v1` (upstream mcp-server example), `http://10.0.0.5:8080/mcp` (a private-looking IPv4 literal; not a guarantee of operator-internality), and `https://gitlab.internal/org/repo` pass **because** they are absolute `http`/`https`, the same as `https://example.com`. The mapping does not DNS-resolve, fetch, or otherwise dereference any URL. Catalog click-through and later source-location fetches are ingestion/runtime concerns; operators must trust the configured registry. A TEST-NET address (RFC 5737, e.g. `192.0.2.1`) may be used in docs as a reserved example that will not collide with a real VPC; it is not RFC 1918 and is not required by this mapping. + +**Alternatives considered:** (a) Denylist only `javascript:`/`data:` — rejected; `file:`/`vbscript:`/`blob:` remain executable or local-file vectors in a catalog UI. (b) Fail the whole mapping on a bad scheme — rejected; one poisoned `websiteUrl` must not drop an otherwise valid server. (c) Block hosts that look private or loopback — rejected; that would require a host classifier this transform does not have, would break the upstream `http://localhost` example, and would still not guarantee operator-internality. (d) Allowlist plus fetch/HEAD to verify the URL — rejected; the transform is side-effect-free (D6) and must not become an SSRF client. (e) Keep a refused `repository.url` on `modelcontextprotocol.io/repository.url` for round-trip — rejected; that annotation is URL-shaped and consumers may treat it as a link. (f) Re-project a refused URL under a different key — rejected; it would still publish `javascript:`/`data:` into searchable annotations. (g) Fall through refused `websiteUrl` / `remotes[].url` to generic projection — rejected; those annotations are still links. + +**Rationale:** Emitted URL fields and projected URL annotations are both catalog-visible, searchable strings that consumers may treat as hyperlinks. An allowlist on **every** such emission is the smallest rule that keeps `javascript:`/`data:` (and cousins) out of the entity. Host appearance is out of scope because this is a pure function of registry JSON, not a browser, crawler, or network classifier. `data:` icon URIs are omitted as a consequence of the same allowlist. + +## Risks / Trade-offs + +- **63-char truncation collisions** → Deterministic hash suffix on truncation and on sanitization collisions keeps keys unique; the hash is derived from the full source path so it is stable across runs. +- **`metadata.name` collisions across registries** (same name+version from two registries under the default prefix) → Out of scope here (no dedup). The caller-overridable prefix is the ingestion-layer lever for per-source namespacing; documented so the future ingestion change can supply distinct prefixes or otherwise dedup. Within a single `(prefix, name, version)` the per-input hash-suffix rule (lossy sanitization or truncation) keeps that identity stable and distinct from a different unsanitized triple that happens to share a sanitized stem. +- **Draft schema drift** → D7 fail-open projection; the mapping table is versioned against the draft and revisited when the schema changes. +- **Lossy flattening of deep `packages[]` config** → Accepted; runtime package details are preserved as scalar-leaf annotations for discoverability, not interpreted. Round-trip fidelity is guaranteed only for scalar leaves. +- **Secret leakage into searchable annotations** (remote `headers`/`variables`, `environmentVariables` carrying `default`/`value`/`choices`) → D9 prunes the `default`/`value`/`choices` leaves of any `isSecret: true` input from projection. This is a deliberate carve-out from scalar round-trip fidelity — those leaves are intentionally unrecoverable from the entity. Non-secret metadata on the same input still projects, so discoverability is preserved. +- **Upstream shape may change** (RFC #32062 could reintroduce `spec.definition` or formalize `spec.remotes` in the base schema) → The target shape is isolated to `mcp-registry-server-mapping`; a shape change is a localized spec/mapping update. +- **Unknown or exotic SCM browse URLs** → Hosts not in the `github`/`gitlab`/`bitbucket`/`azure-devops` template set get a path-join fallback that may 404 as a web browse URL; `repository.source` and `repository.subfolder` are still projected so a later ingestion change can specialize. `HEAD` is used instead of guessing `main`; a host that does not honor `HEAD` in browse URLs still has a deterministic, ref-correct target. +- **Catalog UI click-through to `http`/`https` URLs** → D11 copies every passing `http`/`https` URL, including hosts that look private (`localhost`, `10.0.0.5`, `gitlab.internal`). The mapping does not know or guarantee internality and never fetches those URLs; a later provider or the catalog UI might. Operators must treat the configured registry as trusted. `javascript:`/`data:` never land in native URL fields **or** projected annotations. D11-refused URL scalars are a round-trip exception. + +## Migration Plan + +Not applicable — new capabilities with no existing data or behavior to migrate. The mapping is additive and has no runtime deployment surface of its own until a future ingestion change consumes it. + +## Open Questions + +- Should the mapping optionally map the reverse-DNS namespace (portion before `/` in `server.json` `name`) to `metadata.namespace`, or keep a single default namespace? Deferred to the ingestion change, where entity-ref implications are clearer. Cross-registry uniqueness can already be approached via a per-source prefix override (D4) without introducing `metadata.namespace`. +- Cross-registry dedup/merge of the same server (same name+version from multiple registries) — deferred to the ingestion change. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/journal.jsonl b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/journal.jsonl new file mode 100644 index 00000000000..36e13267400 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/journal.jsonl @@ -0,0 +1,106 @@ +{"ts":"2026-08-18T14:25:44Z","phase":"proposal","event":"change.created","ref":"mcp-registry-server-mapping","output":"Scaffolded change for mapping MCP registry server.json entries to catalog API mcp-server entities"} +{"ts":"2026-08-18T14:25:44Z","phase":"any","event":"turn.start","input":"Create data mapping: MCP registry server.json draft schema to catalog API mcp-server entities; unmapped attrs to modelcontextprotocol.io annotations"} +{"ts":"2026-08-18T14:55:24Z","phase":"authoring","event":"artifact.added","ref":"proposal.md","output":"Proposal: server.json to mcp-server API entity mapping; target uses top-level spec.remotes (no spec.definition); unmapped attrs to modelcontextprotocol.io annotations"} +{"ts":"2026-08-18T15:57:35Z","phase":"authoring","event":"artifact.added","ref":"specs/mcp-registry-server-mapping/spec.md","output":"Direct mapping spec: server.json to mcp-server API entity; metadata.name=__; remotes to spec.remotes; owner/lifecycle supply; deterministic"} +{"ts":"2026-08-18T15:57:35Z","phase":"authoring","event":"artifact.added","ref":"specs/mcp-registry-annotation-projection/spec.md","output":"Annotation projection spec: dot-separated modelcontextprotocol.io keys, sanitization/63-char truncation, no-overwrite, scalar round-trip"} +{"ts":"2026-08-18T15:57:46Z","phase":"any","event":"decision","input":"Resolve mcp-server entity target shape and version-uniqueness for the mapping","output":"Target top-level spec.remotes (no spec.definition); dot-path modelcontextprotocol.io annotations; metadata.name=__; version to dedicated annotation"} +{"ts":"2026-08-18T16:04:16Z","phase":"authoring","event":"artifact.added","ref":"design.md","output":"Design: target upstream spec.remotes shape; 7 decisions incl dot-path annotations, name__version identity, owner/lifecycle supply, fail-open projection"} +{"ts":"2026-08-18T16:04:16Z","phase":"authoring","event":"artifact.added","ref":"tasks.md","output":"Tasks: mapping reference, direct field mapping, annotation projection, conformance fixtures/verification/docs"} +{"ts":"2026-08-18T16:04:16Z","phase":"any","event":"turn.end","output":"Created change mcp-registry-server-mapping with proposal, 2 specs, design, tasks; target spec.remotes (no definition), name__version identity, dot-path annotations"} +{"ts":"2026-08-21T22:33:53Z","phase":"any","event":"turn.start","input":"Remove status field mapping - not in base server.schema.json spec"} +{"ts":"2026-08-21T22:58:20Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/, design.md, tasks.md","input":"Remove status field - not in base server.schema.json per user finding","output":"Removed status→lifecycle derivation; lifecycle now defaults to production (caller-overridable)"} +{"ts":"2026-08-21T22:58:32Z","phase":"any","event":"turn.end","output":"Updated all artifacts to remove status field mapping (not in base schema); spec.lifecycle now defaults to production"} +{"ts":"2026-08-21T23:06:30Z","phase":"any","event":"turn.start","input":"Run pre-implementation audit on updated artifacts"} +{"ts":"2026-08-21T23:20:59Z","phase":"any","event":"turn.end","output":"Audit complete: all artifacts coherent, ready for implementation"} +{"ts":"2026-08-21T23:21:13Z","phase":"any","event":"skill.invoked","name":"openspec-audit-change"} +{"ts":"2026-08-24T15:33:00Z","phase":"any","event":"turn.start","input":"Revise repository.url/websiteUrl link mapping: repository.url→both backstage.io/source-location and metadata.links; link titles Website/Source Code"} +{"ts":"2026-08-24T15:33:00Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/, design.md, tasks.md","input":"repository.url maps to both backstage.io/source-location and a metadata.links entry; websiteUrl link titled Website; repository.url link titled Source Code","output":"Reversed prior 'no source-location' rule; added D10; retitled homepage/source links to Website/Source Code across all artifacts"} +{"ts":"2026-08-24T15:33:00Z","phase":"any","event":"turn.end","output":"Updated all artifacts: repository.url now dual-emits source-location + metadata.links (Source Code); websiteUrl link titled Website"} +{"ts":"2026-08-25T00:03:43Z","phase":"any","event":"turn.start","input":"Re-run pre-implementation audit after repository.url/websiteUrl link-mapping revision"} +{"ts":"2026-08-25T00:03:43Z","phase":"any","event":"skill.invoked","name":"openspec-audit-change"} +{"ts":"2026-08-25T00:03:43Z","phase":"any","event":"turn.end","output":"Audit clean (no CRITICAL). 2 passes; pass1 1 WARNING fixed (remotes type/url in projection scenario), pass2 2 SUGGESTIONs fixed (proposal wording alignment). Artifacts coherent."} +{"ts":"2026-09-11T17:15:00Z","phase":"any","event":"turn.start","input":"Define the repository URL combination algorithm explicitly for descriptive-metadata mapping, for compatibility across SCMs"} +{"ts":"2026-09-11T17:15:00Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-server-mapping/spec.md, design.md, proposal.md, tasks.md","input":"PR review: combining repository.url with repository.subfolder was unspecified and the example invented branch main, which is not in server.json; SCM browse paths differ","output":"Specified SCM-aware URL combination keyed on repository.source (github/gitlab/bitbucket/azure-devops templates using HEAD, path-join fallback); replaced /tree/main examples; added per-SCM scenarios"} +{"ts":"2026-09-11T17:15:00Z","phase":"any","event":"turn.end","output":"Defined explicit repository URL combination algorithm across mapping OpenSpec artifacts; no invented default branch"} + +{"ts":"2026-09-14T13:58:55Z","phase":"any","event":"turn.start","input":"Continue mcp-registry-server-mapping; metadata.name uses ____ with default prefix mcp.registry, caller MAY override."} +{"ts":"2026-09-14T13:58:55Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T14:01:02Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/, design.md, tasks.md","input":"metadata.name should use ____; prefix defaults to mcp.registry, caller MAY override","output":"Identity is prefix__name__version; default prefix mcp.registry; empty prefix falls back; hash includes prefix"} +{"ts":"2026-09-14T14:01:02Z","phase":"any","event":"decision","input":"Resolve metadata.name identity format after all artifacts existed","output":"D4 uses ____; prefix constant mcp.registry by default, caller MAY override; empty sanitizes to default"} +{"ts":"2026-09-14T14:01:02Z","phase":"any","event":"turn.end","output":"Revised all artifacts: metadata.name is ____ with default prefix mcp.registry"} +{"ts":"2026-09-14T15:41:19Z","phase":"any","event":"turn.start","input":"Audit mcp-registry-server-mapping artifacts for pre-implementation coherence"} +{"ts":"2026-09-14T15:41:19Z","phase":"any","event":"skill.invoked","name":"openspec-audit-change"} +{"ts":"2026-09-14T15:42:10Z","phase":"any","event":"agent.spawned","kind":"audit","count":"1"} +{"ts":"2026-09-14T15:50:44Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md, design.md, .openspec.yaml","input":"Apply pass-1 audit autofix batch (SCM templates, url: source-location, reserved keys, tags wording, schema, D-note)","output":"Propagated D10 into tasks; schema rhdh-spec-driven; D9 points at projection spec only"} +{"ts":"2026-09-14T15:51:07Z","phase":"any","event":"agent.spawned","kind":"audit","count":"1"} +{"ts":"2026-09-14T15:59:34Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md","input":"Apply pass-2 audit autofix: project repository.subfolder; hash suffix from full source path","output":"Task 3.5 lists repository.subfolder; task 3.2 hash suffix derived from full source path"} +{"ts":"2026-09-14T15:59:58Z","phase":"any","event":"agent.spawned","kind":"audit","count":"1"} +{"ts":"2026-09-14T16:09:46Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md","input":"Pass-3 autofix: modelcontextprotocol.io/name and /version are annotations not fields","output":"Proposal now calls dedicated name/version annotations, matching D4"} +{"ts":"2026-09-14T16:09:46Z","phase":"any","event":"turn.end","output":"Audit clean (no CRITICAL). 5 WARNING, 8 SUGGESTION. Autofix batches applied; remaining findings are judgment calls."} +{"ts":"2026-09-14T17:07:47Z","phase":"any","event":"turn.start","input":"Continue mcp-registry-server-mapping; name mcp-registry-provider as owner of ingestion non-goal in proposal.md:29"} +{"ts":"2026-09-14T17:07:47Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T17:07:51Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md","input":"Revise ingestion non-goal to name mcp-registry-provider as the owner","output":"Proposal Non-goals now points ingestion/runtime at sibling mcp-registry-provider"} +{"ts":"2026-09-14T17:07:51Z","phase":"any","event":"turn.end","output":"Revised proposal.md ingestion non-goal to name mcp-registry-provider; all artifacts still complete"} +{"ts":"2026-09-14T17:17:16Z","phase":"any","event":"turn.start","input":"Continue mapping: hash metadata.name when sanitization mutates identity or the value is truncated"} +{"ts":"2026-09-14T17:17:16Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T17:18:11Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-server-mapping/spec.md, design.md, tasks.md","input":"Hash metadata.name when sanitization changes identity or the value is truncated","output":"Per-input hash on lossy sanitization or truncation; no two-document collision WHEN"} +{"ts":"2026-09-14T17:18:11Z","phase":"any","event":"turn.end","output":"Identity hash is per-input: sanitization mutated the identity or truncated over-length names"} +{"ts":"2026-09-14T18:06:51Z","phase":"any","event":"turn.start","input":"Direct-map original repository.url to modelcontextprotocol.io/repository.url without normalizing, for round-trip"} +{"ts":"2026-09-14T18:06:51Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T18:08:57Z","phase":"any","event":"decision","input":"Preserve original repository.url for round-trip after D10 base normalization","output":"Dedicated modelcontextprotocol.io/repository.url is the unnormalized source scalar; combination still strips .git and trailing slash"} +{"ts":"2026-09-14T18:08:57Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md","input":"Direct-map original repository.url without normalizing","output":"Proposal names dedicated unnormalized modelcontextprotocol.io/repository.url"} +{"ts":"2026-09-14T18:08:57Z","phase":"authoring","event":"artifact.revised","ref":"design.md","input":"Direct-map original repository.url without normalizing","output":"D2 and D10 copy original URL verbatim; combination base is a separate copy"} +{"ts":"2026-09-14T18:08:57Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-server-mapping/spec.md","input":"Direct-map original repository.url without normalizing","output":"Requirement plus scenario: original scalar on dedicated annotation"} +{"ts":"2026-09-14T18:08:57Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-annotation-projection/spec.md","input":"Direct-map original repository.url without normalizing","output":"Skip re-projecting repository.url; round-trip via dedicated annotation"} +{"ts":"2026-09-14T18:08:57Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md","input":"Direct-map original repository.url without normalizing","output":"Tasks 1.2 2.3 3.4 4.1 cover dedicated unnormalized repository.url"} +{"ts":"2026-09-14T18:08:57Z","phase":"any","event":"turn.end","output":"Revised mapping artifacts: dedicated unnormalized modelcontextprotocol.io/repository.url; D10 still normalizes combination only"} +{"ts":"2026-09-14T18:26:27Z","phase":"any","event":"turn.start","input":"Disallow javascript/data schemes; clarify internal-network URL policy before copying into link fields"} +{"ts":"2026-09-14T18:26:27Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T18:29:41Z","phase":"any","event":"decision","input":"Scheme allowlist vs internal-network blocking for catalog link fields","output":"Allowlist http/https on links, source-location, remotes; allow internal/localhost; never fetch; javascript/data omitted from those fields"} +{"ts":"2026-09-14T18:29:41Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md","input":"Disallow javascript/data; clarify internal-network URL policy","output":"Proposal states http/https allowlist; internal http(s) allowed; no fetch"} +{"ts":"2026-09-14T18:29:41Z","phase":"authoring","event":"artifact.revised","ref":"design.md","input":"Disallow javascript/data; clarify internal-network URL policy","output":"Added D11 allowlist; D8/D10 gated; internal http(s) allowed"} +{"ts":"2026-09-14T18:29:41Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-server-mapping/spec.md","input":"Disallow javascript/data; clarify internal-network URL policy","output":"Emitted URL scheme policy plus javascript/data and localhost scenarios"} +{"ts":"2026-09-14T18:29:41Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-annotation-projection/spec.md","input":"Disallow javascript/data; clarify internal-network URL policy","output":"Refused URLs fall through to projection for round-trip"} +{"ts":"2026-09-14T18:29:41Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md","input":"Disallow javascript/data; clarify internal-network URL policy","output":"Tasks 1.5 2.7 4.7 plus D11 gates on 2.3 2.4 3.4"} +{"ts":"2026-09-14T18:29:41Z","phase":"any","event":"turn.end","output":"Revised mapping artifacts with D11: http/https allowlist, javascript/data refused, internal URLs allowed, no fetch"} +{"ts":"2026-09-14T18:39:27Z","phase":"any","event":"turn.start","input":"Apply D11 allowlist to modelcontextprotocol.io/repository.url as well"} +{"ts":"2026-09-14T18:39:27Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T18:41:41Z","phase":"any","event":"decision","input":"Should modelcontextprotocol.io/repository.url bypass D11 for round-trip?","output":"No: dedicated annotation is an emitted URL field; refused repository.url is a round-trip exception and is not re-projected onto that key"} +{"ts":"2026-09-14T18:41:41Z","phase":"authoring","event":"artifact.revised","ref":"design.md","input":"Apply D11 to modelcontextprotocol.io/repository.url","output":"D11 list includes dedicated annotation; D10/D2/goals match"} +{"ts":"2026-09-14T18:41:41Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-server-mapping/spec.md","input":"Apply D11 to modelcontextprotocol.io/repository.url","output":"Emitted URL fields include dedicated annotation; data: and scp omit it"} +{"ts":"2026-09-14T18:41:41Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-annotation-projection/spec.md","input":"Apply D11 to modelcontextprotocol.io/repository.url","output":"Do not re-project refused repository.url onto dedicated key; round-trip exception"} +{"ts":"2026-09-14T18:41:41Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md","input":"Apply D11 to modelcontextprotocol.io/repository.url","output":"Proposal emitted URL fields include dedicated annotation"} +{"ts":"2026-09-14T18:41:41Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md","input":"Apply D11 to modelcontextprotocol.io/repository.url","output":"Tasks 1.5 2.3 2.7 3.4 4.7 gate dedicated repository.url"} +{"ts":"2026-09-14T18:41:41Z","phase":"any","event":"turn.end","output":"D11 now gates modelcontextprotocol.io/repository.url; refused repository.url is not re-projected onto that key"} +{"ts":"2026-09-14T18:46:47Z","phase":"any","event":"turn.start","input":"Do not project any links that fail D11, including websiteUrl and remotes.url fall-through"} +{"ts":"2026-09-14T18:46:47Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T18:49:35Z","phase":"any","event":"decision","input":"Project refused websiteUrl and remotes.url as annotations?","output":"No: D11 applies to every projected URL; refused links are omitted from native fields and annotations"} +{"ts":"2026-09-14T18:49:35Z","phase":"authoring","event":"artifact.revised","ref":"design.md","input":"Do not project any links that fail D11","output":"D11 covers native and projected URLs; no fall-through"} +{"ts":"2026-09-14T18:49:35Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-server-mapping/spec.md","input":"Do not project any links that fail D11","output":"Scheme policy gates projection of websiteUrl remotes.url icons.src"} +{"ts":"2026-09-14T18:49:35Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-annotation-projection/spec.md","input":"Do not project any links that fail D11","output":"Projection omits all D11-refused URLs; round-trip exception expanded"} +{"ts":"2026-09-14T18:49:35Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md","input":"Do not project any links that fail D11","output":"Proposal: refused URLs not projected under any key"} +{"ts":"2026-09-14T18:49:35Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md","input":"Do not project any links that fail D11","output":"Tasks 1.5 2.4 3.4 3.5 4.7 omit projecting refused URLs"} +{"ts":"2026-09-14T18:49:35Z","phase":"any","event":"turn.end","output":"D11 now omits refused URLs from native fields and from all annotation projection"} +{"ts":"2026-09-14T18:59:48Z","phase":"any","event":"turn.start","input":"Grill: is 10.0.0.5 guaranteed to be an internal IP?"} +{"ts":"2026-09-14T18:59:48Z","phase":"any","event":"skill.invoked","name":"grilling"} +{"ts":"2026-09-14T18:59:53Z","phase":"any","event":"turn.end","output":"Fact: 10.0.0.5 is RFC1918-reserved, not guaranteed operator-internal. First grill: reword spec example vs keep as-is"} +{"ts":"2026-09-14T19:02:56Z","phase":"any","event":"turn.start","input":"Drop RFC1918/internal causal language; keep 10.0.0.5 as http fixture only"} +{"ts":"2026-09-14T19:02:56Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T19:04:04Z","phase":"any","event":"decision","input":"Keep 10.0.0.5 as http fixture; drop RFC1918 causal language","output":"Host appearance is not a scheme-gate input; TEST-NET noted as optional docs example, not RFC 1918"} +{"ts":"2026-09-14T19:04:04Z","phase":"authoring","event":"artifact.revised","ref":"design.md","input":"Drop RFC1918/internal causal language","output":"D11: no host classification; 10.0.0.5 is an http fixture only"} +{"ts":"2026-09-14T19:04:04Z","phase":"authoring","event":"artifact.revised","ref":"specs/mcp-registry-server-mapping/spec.md","input":"Drop RFC1918/internal causal language","output":"Scenarios copy localhost/10.0.0.5/gitlab.internal because scheme is http(s)"} +{"ts":"2026-09-14T19:04:04Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md","input":"Drop RFC1918/internal causal language","output":"Proposal: private-looking hosts pass the same scheme gate"} +{"ts":"2026-09-14T19:04:04Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md","input":"Drop RFC1918/internal causal language","output":"Tasks 1.5 2.7 4.7 no longer treat RFC1918 as a classifier"} +{"ts":"2026-09-14T19:04:04Z","phase":"any","event":"turn.end","output":"Reworded D11: 10.0.0.5 is an http fixture; transform does not guarantee internality"} +{"ts":"2026-09-14T19:13:55Z","phase":"any","event":"turn.start","input":"Grill: does D9 match understanding of isSecret vs private registry exposure?"} +{"ts":"2026-09-14T19:13:55Z","phase":"any","event":"skill.invoked","name":"grilling"} +{"ts":"2026-09-14T19:13:59Z","phase":"any","event":"turn.end","output":"Fact: D9 omits isSecret true default/value only; isSecret false is projected. Asked whether catalog vs registry is the threat model"} +{"ts":"2026-09-14T19:30:35Z","phase":"any","event":"turn.start","input":"Omit choices on isSecret true inputs; align projection spec and task 4.6"} +{"ts":"2026-09-14T19:30:35Z","phase":"any","event":"skill.invoked","name":"openspec-continue-change"} +{"ts":"2026-09-14T19:33:13Z","phase":"authoring","event":"artifact.revised","ref":"proposal.md, specs/mcp-registry-annotation-projection/spec.md, design.md, tasks.md","input":"Redact choices on isSecret true inputs; keep the same rule in the projection spec and task 4.6","output":"D9 now prunes default/value/choices; placeholder still projects; rejected documenting that choices cannot be secrets"} +{"ts":"2026-09-14T19:33:13Z","phase":"any","event":"decision","input":"Audit H: redact choices on isSecret true, or document why choices cannot contain secrets","output":"Omit choices with default/value; MCP Input.choices is a list of possible values, so it is value-bearing"} +{"ts":"2026-09-14T19:33:13Z","phase":"any","event":"turn.end","output":"Revised D9: isSecret true prunes default/value/choices; projection spec and task 4.6 match; artifacts still complete"} +{"ts":"2026-09-14T21:05:28Z","phase":"any","event":"turn.start","input":"Audit mcp-registry-server-mapping artifacts for pre-implementation coherence"} +{"ts":"2026-09-14T21:05:28Z","phase":"any","event":"skill.invoked","name":"openspec-audit-change"} +{"ts":"2026-09-14T21:15:33Z","phase":"authoring","event":"artifact.revised","ref":"tasks.md, specs/mcp-registry-annotation-projection/spec.md","input":"Apply pass-1 audit autofix batch (D11 refuse set, reserved source-location, URL-typed leaf, remotes [] fixtures)","output":"1.5/2.7 list vbscript blob //host; reserved examples include source-location; 3.4 D11 membership; 4.1 empty remotes fixtures"} +{"ts":"2026-09-14T21:28:42Z","phase":"any","event":"turn.end","output":"Audit clean (no CRITICAL). Pass-1 autofix applied; pass-2 schema rename skipped. Remaining: skip-or-disambiguate, null-omit MAY, sanitization, .openspec.yaml, proposal length."} diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/proposal.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/proposal.md new file mode 100644 index 00000000000..1a7ec6e6e1f --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/proposal.md @@ -0,0 +1,77 @@ +# Proposal: MCP Registry Server Mapping + +## Why + +MCP registries (the upstream [MCP Registry](https://github.com/modelcontextprotocol/registry) and downstream/private mirrors) publish server entries as `server.json` documents conforming to the [`server.schema.json` draft](https://raw.githubusercontent.com/modelcontextprotocol/registry/refs/heads/main/docs/reference/server-json/draft/server.schema.json). Backstage already supports cataloging MCP servers as `API` entities with `spec.type: mcp-server` — defined upstream in [`backstage/backstage`](https://github.com/backstage/backstage) (Backstage RFC [#32062](https://github.com/backstage/backstage/issues/32062), not by RHDH) — but there is no defined, deterministic way to turn a registry `server.json` into such an entity. Without a canonical mapping, every ingestion path would invent its own field translation, lose registry metadata that has no native catalog home, and produce entities that do not round-trip. A single documented mapping contract makes registry ingestion predictable and lossless, and is the prerequisite for any future registry entity provider. + +## What Changes + +- Define a deterministic, idempotent **mapping contract** that transforms one `server.json` document (draft `server.schema.json`) into one Backstage `API` entity with `spec.type: mcp-server`, conformant to the upstream Backstage `mcp-server` `API` entity specification (defined in [`backstage/backstage`](https://github.com/backstage/backstage)). +- Specify the **direct field mapping**: which `server.json` attributes lift into native Backstage/entity fields — `metadata.name` (derived as `____` so each server version gets a collision-free entity; `prefix` is the constant `mcp.registry` by default and a caller MAY supply an override), `metadata.title`/`description`, `metadata.links` (a `websiteUrl` entry titled `Website` and a `repository.url` entry titled `Source Code`, with `repository.url` combined with `repository.subfolder` when present via the SCM-aware algorithm in `mcp-registry-server-mapping` — `repository.source` selects a browse-path template and `HEAD` stands in for the unspecified default branch, e.g. `url:https://github.com/org/repo/tree/HEAD/subfolder` — also emitted as a `backstage.io/source-location` annotation whose value MUST use the `url:` format), a dedicated unnormalized `modelcontextprotocol.io/repository.url` annotation holding the original `repository.url` scalar, `spec.type`/`lifecycle`/`owner`, the top-level `spec.remotes[]` (`type`, `url`) that the upstream mcp-server entity uses in place of `spec.definition`, and dedicated `modelcontextprotocol.io/name` and `modelcontextprotocol.io/version` annotations preserving the bare canonical name and version. **Emitted URL fields** (`metadata.links[].url`, `backstage.io/source-location`, `spec.remotes[].url`, `modelcontextprotocol.io/repository.url`) and **projected URL annotations** copy only absolute `http`/`https` URLs — `javascript:`/`data:` and other refused schemes are omitted from those fields and are not projected under any key; hosts that look private (`localhost`, `10.0.0.5`, `gitlab.internal`) pass the same scheme gate (the transform does not classify or guarantee internality) because it never fetches URLs. +- Specify the **annotation projection** fallback: every `server.json` attribute that has no native home is projected into an annotation keyed `modelcontextprotocol.io/`, encoding nested objects and array indices as dot-separated segments within a single-slash, catalog-valid key. **Secret-flagged inputs** (`isSecret: true`) have their `default`/`value`/`choices` leaves pruned so no credential is published into a searchable catalog annotation. **D11-refused URLs** are not projected. +- Specify **catalog-valid key construction**: character sanitization, the 63-character name-segment limit with deterministic hash-suffix truncation, and collision disambiguation — because Backstage annotation keys permit only one `/` and a restricted name-segment character set. +- Specify **field-supply rules** for catalog-required fields absent from `server.json` (`spec.owner` defaults to the constant `unknown`, `spec.lifecycle` defaults to the constant `production`, identity `prefix` defaults to the constant `mcp.registry` — all overridable by caller defaults, never a failure), and **round-trip fidelity** so scalar leaves are recoverable from the produced entity (except redacted secrets and D11-refused URLs). +- Deliver a canonical mapping-table reference, worked examples, and a conformance fixture set (`server.json` input → expected entity output). + +## Capabilities + +### New Capabilities + +- `mcp-registry-server-mapping`: The deterministic direct field mapping from a `server.json` (draft `server.schema.json`) document to the native fields of an `mcp-server` `API` entity — identity/name sanitization as `____` with canonical-name preservation, descriptive metadata including a dedicated unnormalized `modelcontextprotocol.io/repository.url`, `remotes` → top-level `spec.remotes[]`, emitted-URL scheme allowlist (`http`/`https` only; no host classification), and supply of catalog-required fields (`spec.owner`, `spec.lifecycle`) and the identity prefix absent from the source. +- `mcp-registry-annotation-projection`: The fallback rule that projects every `server.json` attribute without a native home into `modelcontextprotocol.io/` annotations — dot-path encoding of nested objects and array indices, catalog-valid key sanitization/truncation, no-overwrite of natively-mapped fields, D9 redaction of `isSecret: true` `default`/`value`/`choices` leaves, D11 URL allowlist on projected links, and scalar round-trip fidelity. + +### Modified Capabilities + +_(none — no long-lived specs exist under `openspec/specs/` yet; this change introduces new capabilities only and consumes the upstream Backstage `mcp-server` `API` entity contract defined in [`backstage/backstage`](https://github.com/backstage/backstage).)_ + +## Non-goals + +- **Ingestion / runtime.** No registry HTTP client, polling schedule, catalog entity provider/processor, or entity lifecycle management. This change defines the pure `server.json` → entity transform only; fetching entries and applying the mapping is owned by [`mcp-registry-provider`](../mcp-registry-provider/). +- **Modifying the upstream `mcp-server` `API` entity contract.** This change consumes that contract as defined in [`backstage/backstage`](https://github.com/backstage/backstage); it does not change `spec.type: mcp-server` recognition, validation, or page rendering. Changes to the entity contract are owned upstream (Backstage RFC [#32062](https://github.com/backstage/backstage/issues/32062)), not made here. +- **Reverse mapping** (entity → `server.json`) beyond the scalar round-trip fidelity guarantee needed to avoid data loss. +- **Runtime invocation / health checking** of the mapped MCP servers. +- **Local package execution.** `packages[]` runtime details are preserved as annotations for discoverability, not interpreted or executed. +- **Deduplication / merge** of the same server appearing in multiple registries. + +## Upstream References + +The mapping target — the `mcp-server` `API` entity shape — is anchored to the following upstream Backstage sources (`backstage/backstage`): + +- **Canonical example** — [`packages/catalog-model/examples/apis/backstage-mcp-server-api.yaml`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/examples/apis/backstage-mcp-server-api.yaml): the reference `mcp-server` `API` entity. It uses `spec.type: mcp-server`, `spec.lifecycle`, `spec.owner`, `metadata.tags: [mcp, ai]`, and — notably — a **top-level `spec.remotes[]`** with `type` + `url` (no `spec.definition`): + ```yaml + apiVersion: backstage.io/v1alpha1 + kind: API + metadata: + name: backstage-mcp-server + description: An MCP server that exposes tools related to the Backstage ecosystem + tags: [mcp, ai] + spec: + type: mcp-server + lifecycle: experimental + owner: team-a + remotes: + - type: streamable-http + url: http://localhost:7007/api/mcp/v1 + ``` +- **Base API schema** — [`packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json): defines `spec.type` (free-form string), `lifecycle`, `owner`, and lists `definition` as required for generic `API` entities. This `definition` requirement is **overridden for `spec.type: mcp-server`** by the dedicated mcp-server entity schema below. +- **RFC** — Backstage [#32062](https://github.com/backstage/backstage/issues/32062): models MCP servers as the `API` kind with `spec.type: mcp-server`. +- **Dedicated mcp-server entity schema** — [`packages/catalog-model/src/kinds/McpServerApiEntity.ts`](https://github.com/backstage/backstage/blob/f91434377dc43cd64bef82344e3f2b539bfdaf11/packages/catalog-model/src/kinds/McpServerApiEntity.ts#L28-L36) (Backstage [PR #34016](https://github.com/backstage/backstage/pull/34016)): the upstream schema for `spec.type: mcp-server`. It does `Omit` and requires `spec.type: 'mcp-server'`, `spec.lifecycle`, `spec.owner`, and `spec.remotes[]` (with optional `spec.system`); `spec.definition` is not part of this schema. `spec.remotes` therefore **replaces** the base schema's required `spec.definition` for mcp-server entities. + +**Consequence for this mapping:** the mapping target is the upstream example shape exactly — `spec.type: mcp-server`, `spec.lifecycle`, `spec.owner`, `metadata` (`name`/`title`/`description`/`tags`/`links`), and top-level `spec.remotes[]` (`type`, `url`). No `spec.definition` is emitted. `server.json` fields with no native home (`packages`, `repository` sub-fields other than `url`, `icons`, remote `headers`/`variables`, `_meta`, …) are projected into `modelcontextprotocol.io/*` annotations per `mcp-registry-annotation-projection`. (`name` and `version` get dedicated `modelcontextprotocol.io/name` and `modelcontextprotocol.io/version` annotations and, with the identity prefix, also form `metadata.name`. When it passes D11, the original `repository.url` is copied unnormalized into dedicated `modelcontextprotocol.io/repository.url`; the combined browse URL is a separate `backstage.io/source-location` / Source Code link.) + +## Canonical Touchpoints + +- **PRDs (`specifications/prd/`)**: None +- **ADRs (`specifications/adr/`)**: None +- **Long-lived specs (`openspec/specs/`)**: None (new capabilities only; `openspec/specs/` does not yet exist) + +**Change type**: feature-spec + +## Impact + +- **Depends on the upstream Backstage `mcp-server` `API` entity specification** (defined in [`backstage/backstage`](https://github.com/backstage/backstage), not by RHDH): the mapping targets that `mcp-server` `API` entity contract — `spec.type: mcp-server`, `spec.lifecycle`, `spec.owner`, and top-level `spec.remotes[]` (`type`, `url`); no `spec.definition`. Produced entities MUST pass the upstream `mcp-server` `API` entity schema (`McpServerApiEntity`, PR [#34016](https://github.com/backstage/backstage/pull/34016)), which for `spec.type: mcp-server` requires `spec.remotes` and drops the base schema's `spec.definition` requirement. +- **Source schema**: MCP Registry draft `server.schema.json` — top-level `name`, `title`, `description`, `version`, `websiteUrl`, `icons[]`, `repository{url,source,id,subfolder}`, `packages[]` (with nested `runtimeArguments`/`packageArguments`/`environmentVariables`/`transport`), `remotes[]` (`type`, `url`, `headers[]`, `variables`), and `_meta`. The draft schema evolves; the mapping is versioned against the draft and projects unknown fields generically (fail-open). +- **Backstage constraints**: annotation keys allow exactly one `/`; the name segment is ≤63 chars over a restricted character set — driving the dot-path encoding, sanitization, and hash-suffix truncation rules. +- **Consumers**: a future registry entity provider; AI agents and developers who discover MCP servers via catalog search/filter (the `modelcontextprotocol.io/*` annotations become searchable/filterable metadata). +- **Documentation**: mapping-table reference, worked examples, and a conformance fixture set usable as the test oracle when the transform is implemented. +- **Upstream**: keep aligned with Backstage RFC [#32062](https://github.com/backstage/backstage/issues/32062) and the MCP Registry `server.json` draft as both evolve. diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-annotation-projection/spec.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-annotation-projection/spec.md new file mode 100644 index 00000000000..41601fda99e --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-annotation-projection/spec.md @@ -0,0 +1,138 @@ +## MCP Registry Annotation Projection + +This capability defines the fallback that captures every `server.json` attribute which has no native home in the `mcp-server` `API` entity shape (see `mcp-registry-server-mapping`). Such attributes are projected into entity annotations under the `modelcontextprotocol.io/` prefix, keyed by the attribute's path within the source document. + +Attribute paths are encoded in **dot-separated** form as `modelcontextprotocol.io/attribute.tree.to.leaf`. Because Backstage annotation keys permit exactly one `/` (separating the DNS-style prefix from the name segment) and limit the name segment to a restricted character set and 63 characters, the name segment is sanitized and length-bounded so every produced key is catalog-valid. + +--- + +## ADDED Requirements + +### Requirement: Project unmapped server.json attributes into modelcontextprotocol.io annotations + +Every scalar leaf in the `server.json` document that is not consumed by a native field mapping SHALL be projected into an entity annotation whose key is `modelcontextprotocol.io/`, where `` identifies the attribute's location in the source document. Attributes that the direct mapping already places in a native field or a dedicated annotation SHALL NOT be re-projected by this fallback. Projection SHALL NOT emit a URL that fails the emitted-URL scheme policy (D11) under any key — including refused `websiteUrl`, `repository.url`, `remotes[].url`, `icons[].src`, and any other URL-typed leaf. Non-URL siblings of a refused URL (for example `remotes[].type`, `icons[].mimeType`, `repository.source`) SHALL still project. + +#### Scenario: Unmapped scalar becomes an annotation + +- **WHEN** a `server.json` carries a scalar with no native home, such as `icons[0].mimeType: image/png` +- **THEN** the entity has an annotation `modelcontextprotocol.io/icons.0.mimeType` with value `image/png` + +#### Scenario: Natively-mapped attributes are not re-projected + +- **WHEN** a `server.json` carries `remotes[].type`/`url` that were copied into `spec.remotes`, `name` (mapped to `modelcontextprotocol.io/name` and `metadata.name`), `version` (mapped to `modelcontextprotocol.io/version`), `title`/`description` (mapped to `metadata`), `websiteUrl` that was copied into `metadata.links`, and `repository.url` that was copied into `metadata.links`, `backstage.io/source-location`, and the dedicated `modelcontextprotocol.io/repository.url` annotation +- **THEN** those attributes are not additionally emitted as generic `modelcontextprotocol.io/*` projected annotations, and the generic projection does not overwrite or re-derive the direct-mapping `backstage.io/source-location` or `modelcontextprotocol.io/repository.url` annotations + +#### Scenario: Refused URLs are not projected + +- **WHEN** `websiteUrl`, `remotes[].url`, `repository.url`, or `icons[].src` is refused by the emitted-URL scheme policy (for example `javascript:` or `data:`) +- **THEN** no `modelcontextprotocol.io/*` annotation carries that URL, and non-URL siblings of the same object still project + +#### Scenario: Refused repository.url is not re-projected onto the dedicated key + +- **WHEN** `repository.url` is refused by the emitted-URL scheme policy (for example `javascript:`, `data:`, or `git@github.com:org/repo.git`) +- **THEN** no `modelcontextprotocol.io/repository.url` annotation is emitted, generic projection does not recreate that key, and remaining `repository` sub-fields (`source`, `id`, `subfolder`) still project + +### Requirement: Encode nested paths as dot-separated segments within a single-slash key + +Nested object keys and array indices SHALL be encoded as dot-separated segments in the annotation key's name portion, appended to the single `modelcontextprotocol.io/` prefix. Object keys contribute their key name; array elements contribute their zero-based numeric index. Only scalar leaves are emitted as annotation values, serialized as strings; container nodes (objects, arrays) are traversed rather than emitted. + +#### Scenario: Nested object key path + +- **WHEN** a `server.json` carries `repository.source: github` +- **THEN** the entity has an annotation `modelcontextprotocol.io/repository.source` with value `github` + +#### Scenario: Array element index path + +- **WHEN** a `server.json` carries `packages[0].identifier: "@scope/pkg"` +- **THEN** the entity has an annotation `modelcontextprotocol.io/packages.0.identifier` with value `@scope/pkg` + +#### Scenario: Deeply nested leaf under an array + +- **WHEN** a `server.json` carries `remotes[1].headers[0].name: Authorization` +- **THEN** the entity has an annotation `modelcontextprotocol.io/remotes.1.headers.0.name` with value `Authorization` + +#### Scenario: Scalar values are serialized as strings + +- **WHEN** a projected leaf is a non-string scalar such as a boolean or number (e.g. `packages[0].environmentVariables[0].isSecret: true`) +- **THEN** the annotation value is the string form of that scalar (e.g. `true`) + +### Requirement: Produce catalog-valid annotation keys + +Every projected annotation key SHALL be valid for the Backstage catalog: the name segment SHALL contain only allowed characters (alphanumerics plus `-`, `_`, `.`), SHALL begin and end with an alphanumeric character, and SHALL be at most 63 characters. Path segments containing characters outside this set (for example `/`, `$`, `@`, whitespace, or a leading `_` as in `_meta`) SHALL be sanitized deterministically, and keys whose name segment would exceed 63 characters SHALL be truncated and suffixed with a stable hash so they remain valid and unique. + +#### Scenario: Illegal characters in a path are sanitized + +- **WHEN** a `server.json` carries a `_meta` object whose nested key contains a `/` (e.g. `_meta."io.modelcontextprotocol.registry/publisher-provided".x`) +- **THEN** the projected annotation key replaces the leading underscore and the embedded `/` with allowed characters so the resulting `modelcontextprotocol.io/` key is catalog-valid + +#### Scenario: Over-length key is truncated with a stable suffix + +- **WHEN** a projected path's name segment would exceed 63 characters +- **THEN** the mapping truncates it and appends a deterministic hash suffix derived from the full path, keeping the key ≤63 characters + +#### Scenario: Sanitization collisions are disambiguated + +- **WHEN** two distinct source paths sanitize to the same annotation key +- **THEN** the mapping appends a deterministic hash suffix so each source path maps to a distinct key + +### Requirement: Do not overwrite reserved or previously-set annotations + +Projection SHALL NOT overwrite annotations set by the direct mapping (for example `modelcontextprotocol.io/name`, `modelcontextprotocol.io/version`, `modelcontextprotocol.io/repository.url`, `backstage.io/source-location`) or any other reserved annotation. If a generic projected key would collide with such an annotation, the direct-mapping value SHALL win and the projection SHALL be skipped or disambiguated. + +#### Scenario: Direct-mapping annotation wins + +- **WHEN** a generic projection would produce a `modelcontextprotocol.io/name` key that collides with the canonical-name annotation set by the direct mapping +- **THEN** the direct-mapping value is retained and the generic projection does not overwrite it + +### Requirement: Redact secret-flagged input values + +An `Input` object in `server.json` (as used by `packages[].environmentVariables[]`, `remotes[].headers[]`, `remotes[].variables`, and package/runtime arguments) MAY declare `isSecret: true`. When an input object declares `isSecret: true`, the projection SHALL prune (omit) that object's `default`, `value`, and `choices` leaves (including every `choices[]` element) — those values SHALL NOT appear in any `modelcontextprotocol.io/*` annotation. All non-secret sibling leaves of the same input (for example `name`, `description`, `format`, `isRequired`, `isSecret`, `placeholder`) SHALL continue to project normally. This redaction applies uniformly to every `isSecret: true` input regardless of location, not only environment variables. + +#### Scenario: Secret environment variable value is pruned + +- **WHEN** a `server.json` carries `packages[0].environmentVariables[0]` with `isSecret: true` and a populated `default` (or `value`) +- **THEN** no `modelcontextprotocol.io/*` annotation carries that `default`/`value`, while the input's non-secret leaves (e.g. `packages.0.environmentVariables.0.name`, `.isSecret`, `.description`) are still projected + +#### Scenario: Secret input choices are pruned + +- **WHEN** a `server.json` carries an `isSecret: true` input with a populated `choices` array (for example `["tok_live_aaa", "tok_live_bbb"]`) +- **THEN** no `modelcontextprotocol.io/*` annotation carries any `choices` element, while non-secret siblings such as `name` and `isSecret` still project + +#### Scenario: Secret remote header/variable value is pruned + +- **WHEN** a `remotes[].headers[]` or `remotes[].variables` input declares `isSecret: true` with a populated `default`/`value` or `choices` +- **THEN** that `default`/`value`/`choices` is omitted from all annotations, and the redaction behaves identically to the environment-variable case (uniform across input locations) + +#### Scenario: Non-secret input value is retained + +- **WHEN** an input object has `isSecret: false` or omits `isSecret`, with a populated `default`/`value` or `choices` +- **THEN** that `default`/`value`/`choices` is projected into a `modelcontextprotocol.io/*` annotation as normal + +### Requirement: Scalar round-trip fidelity + +Every scalar leaf present in the source `server.json` SHALL be recoverable from the produced entity — either from a native field or from a projected annotation — **except** the `default`/`value`/`choices` leaves of `isSecret: true` inputs, which are intentionally redacted per "Redact secret-flagged input values", **and** any URL refused by the emitted-URL scheme policy, which is omitted from emitted URL fields and from all `modelcontextprotocol.io/*` annotations. Null values and empty containers MAY be omitted per a documented rule; every non-null, non-redacted, non-D11-refused-URL scalar SHALL be represented. + +#### Scenario: All scalar leaves are recoverable + +- **WHEN** a `server.json` with populated `packages`, `repository`, `icons`, and `_meta` is mapped and those URL leaves pass the emitted-URL scheme policy +- **THEN** every non-null, non-redacted, non-D11-refused scalar leaf from those sections is present either in a native entity field or in a `modelcontextprotocol.io/*` annotation, so the source values can be reconstructed + +#### Scenario: Original repository.url is recoverable unnormalized + +- **WHEN** a `server.json` provides `repository.url` `https://github.com/org/repo.git/` +- **THEN** that exact scalar is recoverable from `modelcontextprotocol.io/repository.url`; reconstructing it from the normalized Source Code link or `backstage.io/source-location` is not required and MUST NOT be the only representation + +#### Scenario: Refused URLs are exempt from round-trip + +- **WHEN** a `server.json` provides `websiteUrl` `javascript:alert(1)`, `repository.url` `data:text/html,x`, or `icons[0].src` `javascript:alert(1)` +- **THEN** the absence of those scalars from `metadata.links`, `backstage.io/source-location`, `spec.remotes`, and all `modelcontextprotocol.io/*` annotations does NOT violate round-trip fidelity, because D11 refusal of URL scalars is a documented exception + +#### Scenario: Redacted secret leaves are exempt from round-trip + +- **WHEN** a `server.json` carries an `isSecret: true` input with a populated `default`/`value` or `choices` +- **THEN** the absence of that `default`/`value`/`choices` from the entity does NOT violate round-trip fidelity, because secret redaction is a documented exception + +#### Scenario: Nulls and empty containers follow the documented omission rule + +- **WHEN** a `server.json` attribute is `null` or an empty array/object +- **THEN** it is omitted from the annotations per the documented rule, and its omission does not cause the mapping to fail diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md new file mode 100644 index 00000000000..f9b06139ddb --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/specs/mcp-registry-server-mapping/spec.md @@ -0,0 +1,223 @@ +## MCP Registry Server Mapping + +This capability defines the deterministic, idempotent transform from a single MCP registry `server.json` document (conforming to the draft [`server.schema.json`](https://raw.githubusercontent.com/modelcontextprotocol/registry/refs/heads/main/docs/reference/server-json/draft/server.schema.json)) into a single Backstage `API` entity with `spec.type: mcp-server`. + +The target entity shape follows the upstream Backstage mcp-server example ([`backstage-mcp-server-api.yaml`](https://raw.githubusercontent.com/backstage/backstage/a4bdc49ed664661bc69fe42bfaebcf24dc96e6b3/packages/catalog-model/examples/apis/backstage-mcp-server-api.yaml)): `apiVersion: backstage.io/v1alpha1`, `kind: API`, `metadata` (`name`, `title`, `description`, `tags`, `links`, `annotations`), and `spec` (`type: mcp-server`, `lifecycle`, `owner`, top-level `remotes[]` with `type` + `url`). No `spec.definition` is emitted — top-level `spec.remotes[]` replaces it. + +`server.json` attributes without a native home in this shape are handed to `mcp-registry-annotation-projection`. This spec covers only the direct (native-field) mapping and the supply of catalog-required fields absent from the source. + +--- + +## ADDED Requirements + +### Requirement: Transform a server.json document into a valid mcp-server API entity + +The mapping SHALL transform one `server.json` document into one `API` entity with `apiVersion: backstage.io/v1alpha1`, `kind: API`, and `spec.type: mcp-server`, such that the produced entity passes the upstream Backstage `mcp-server` `API` entity schema (`McpServerApiEntity`), which for `spec.type: mcp-server` requires `spec.remotes` and does not include `spec.definition` — overriding the generic base `API` schema. When the source omits a `server.json`-required field (`name`, `description`, or `version`), the mapping SHALL fail with an actionable error that names the missing field and references the MCP server schema, rather than emitting a partial entity. + +#### Scenario: Minimal valid server.json produces a valid entity + +- **WHEN** the mapping receives a `server.json` with `name`, `description`, `version`, and one `remotes` entry (`type: streamable-http`, valid `url`) +- **THEN** it produces an `API` entity with `kind: API`, `spec.type: mcp-server`, `spec.lifecycle` (`production` when no caller override is supplied), `spec.owner` (`unknown` when no caller override is supplied), and a top-level `spec.remotes` entry carrying that `type` and `url`, and the entity passes the upstream `mcp-server` `API` entity schema (`McpServerApiEntity`) validation + +#### Scenario: Missing required source field fails the mapping + +- **WHEN** the input `server.json` omits a required field such as `name`, `description`, or `version` +- **THEN** the mapping fails with an error that names the missing field and references the MCP server schema, and no entity is produced + +### Requirement: Map remotes to top-level spec.remotes + +The mapping SHALL copy each `server.json` `remotes[]` entry's `type` and `url` into a corresponding top-level `spec.remotes[]` entry on the `API` entity, preserving source order of the entries that are copied. A remote `url` is copied only when it passes the "Emitted URL scheme policy" requirement; a refused remote is omitted from `spec.remotes` (it is not copied with a blank `url`). The refused `url` SHALL NOT be projected; non-URL siblings (`type`, `headers`, `variables`) SHALL still be handed to `mcp-registry-annotation-projection`. When the source `remotes` is empty, unset, or every remote is refused, the mapping SHALL emit an empty `spec.remotes: []` array — never an omitted field — so the output stays deterministic and schema-conformant. The mapping SHALL NOT emit a `spec.definition` field. Remote sub-fields that are not part of the native `spec.remotes` shape (`headers`, `variables`) SHALL be handed to `mcp-registry-annotation-projection` rather than dropped, including on remotes that were copied. + +#### Scenario: Remotes copied in order + +- **WHEN** a `server.json` declares multiple `remotes` entries +- **THEN** `spec.remotes` contains one entry per source remote, in the same order, each with the source `type` and `url`, and no `spec.definition` is present + +#### Scenario: Remote headers and variables are projected, not dropped + +- **WHEN** a `remotes` entry carries `headers` or `variables` +- **THEN** the native `spec.remotes` entry contains only `type` and `url`, and the `headers`/`variables` are projected into `modelcontextprotocol.io/*` annotations keyed by the remote's index + +#### Scenario: Server with no remotes + +- **WHEN** a `server.json` declares no `remotes` (only local `packages`) +- **THEN** the entity is still produced with an empty `spec.remotes: []` (not omitted), remains valid, and the `packages` are projected into annotations + +#### Scenario: Disallowed remote URL scheme is omitted from spec.remotes + +- **WHEN** a `remotes` entry has `url` `javascript:alert(1)` (or another non-`http`/`https` scheme such as `data:`) +- **THEN** that entry is omitted from `spec.remotes`, the mapping still succeeds, the refused `url` is not projected, and non-URL siblings such as `type` are projected into `modelcontextprotocol.io/*` annotations + +#### Scenario: http(s) URLs with private-looking hosts are still copied + +- **WHEN** a `remotes` entry has `url` `http://localhost:7007/api/mcp/v1` or `http://10.0.0.5:8080/mcp` +- **THEN** `spec.remotes` contains that `type` and `url` unchanged; the mapping copies them because the scheme is `http`, not because it classified the host as internal or RFC 1918 + +### Requirement: Derive a version-unique metadata.name and preserve the canonical name and version + +A registry publishes one `server.json` per server version, and each version becomes its own `API` entity; a `metadata.name` derived from the canonical name alone would therefore collide across versions in the catalog. The `server.json` `name` is also a reverse-DNS identifier (`namespace/server`) that is not itself a valid Backstage `metadata.name`. The mapping SHALL derive `metadata.name` as `____` — the sanitized prefix, the sanitized canonical name, and the sanitized version joined by a double underscore (`__`) — conforming to the Backstage name character set (lowercase alphanumerics with `-`/`_`/`.`, beginning and ending alphanumeric, ≤63 characters). The prefix SHALL be the constant `mcp.registry` when no caller override is supplied; a caller MAY supply an override default prefix. When the override is unset, empty, or sanitizes to empty, the mapping SHALL use `mcp.registry` and SHALL NOT fail. After joining, the mapping SHALL append a stable hash suffix derived from the effective prefix, the unmodified canonical name, and the unmodified version whenever **either** sanitization mutates any identity segment (the sanitized segment differs from the source segment) **or** the joined candidate exceeds 63 characters. When the hash is applied, the mapping SHALL truncate the candidate stem as needed so the final `metadata.name` remains ≤63 characters. When sanitization does not mutate any segment and the candidate is ≤63 characters, the mapping SHALL emit the candidate with no hash suffix. This rule is a pure function of one document plus caller defaults; it does not observe other documents. The mapping SHALL preserve the unmodified canonical name (without the version) in a `modelcontextprotocol.io/name` annotation, and SHALL map the `version` directly to its dedicated `modelcontextprotocol.io/version` annotation so it remains individually queryable. + +#### Scenario: Two versions of the same server produce distinct entities + +- **WHEN** two `server.json` documents share the canonical name `io.github.user/weather` but declare `version` `1.0.0` and `2.0.0`, and no caller prefix override is supplied +- **THEN** the two produced entities have distinct `metadata.name` values (each incorporating the default prefix, its version, and a hash suffix because sanitization replaced `/` in the canonical name), each carries the same `modelcontextprotocol.io/name: io.github.user/weather`, and each carries its own `modelcontextprotocol.io/version` (`1.0.0` and `2.0.0` respectively) + +#### Scenario: Reverse-DNS name and version are sanitized into metadata.name with the default prefix + +- **WHEN** `server.json` `name` is `io.github.user/weather`, `version` is `1.0.2`, and no caller prefix override is supplied +- **THEN** `metadata.name` is the sanitized `____` stem `mcp.registry__io.github.user-weather__1.0.2` plus a stable hash suffix (sanitization changed the identity), the original `io.github.user/weather` is preserved verbatim in `modelcontextprotocol.io/name`, and `1.0.2` is recorded in `modelcontextprotocol.io/version` + +#### Scenario: Already catalog-valid identity is emitted without a hash + +- **WHEN** `server.json` `name` is `weather`, `version` is `1.0.2`, and no caller prefix override is supplied +- **THEN** `metadata.name` is `mcp.registry__weather__1.0.2` with no hash suffix (no segment was mutated and the candidate is ≤63 characters) + +#### Scenario: Caller prefix override is applied + +- **WHEN** the mapping is invoked with a caller-provided default prefix `com.example.registry` and `server.json` `name` is `io.github.user/weather` with `version` `1.0.2` +- **THEN** `metadata.name` is the sanitized stem `com.example.registry__io.github.user-weather__1.0.2` plus a stable hash suffix (sanitization changed the identity) + +#### Scenario: Empty prefix override falls back to the default + +- **WHEN** the mapping is invoked with a caller-provided prefix that is empty or sanitizes to empty +- **THEN** `metadata.name` uses the default prefix `mcp.registry` and the mapping succeeds + +#### Scenario: Over-length identity is truncated with a hash + +- **WHEN** the sanitized `____` candidate exceeds 63 characters +- **THEN** the mapping truncates the stem and appends a stable hash suffix derived from the prefix, canonical name, and version so the final `metadata.name` is ≤63 characters + +### Requirement: Map descriptive metadata to native Backstage fields + +The mapping SHALL map `server.json` descriptive attributes to native Backstage `metadata` fields: `title` → `metadata.title`; `description` → `metadata.description`; `websiteUrl` → a `metadata.links` entry whose `url` is the `websiteUrl` and whose `title` is `Website`, **only when** `websiteUrl` passes the "Emitted URL scheme policy" requirement; and `repository.url` combined with `repository.subfolder` when present (per the repository URL combination algorithm below) → a `metadata.links` entry whose `url` is that combined URL and whose `title` is `Source Code`, **only when** the original `repository.url` passes that policy. The mapping SHALL ALSO emit the same combined URL as a `backstage.io/source-location` annotation whose value MUST use the `url:` format (`url:`), so the repository is captured both as the canonical Backstage source-location annotation (for source-aware tooling) and as a human-visible source link — again only when `repository.url` passes the policy. Independently of that combination, when `repository.url` is present **and** passes the policy, the mapping SHALL copy the original `repository.url` scalar **verbatim** (no trailing-`/` or `.git` strip, no subfolder join, no scheme rewrite) into a dedicated `modelcontextprotocol.io/repository.url` annotation so the source value remains recoverable after combination-base normalization. A refused `websiteUrl` SHALL NOT produce a Website `metadata.links` entry and SHALL NOT be projected. A refused `repository.url` SHALL NOT produce a Source Code link, a `backstage.io/source-location` annotation, or a `modelcontextprotocol.io/repository.url` annotation, and SHALL NOT be re-projected onto that annotation key. The mapping SHALL set `metadata.tags` to include the upstream mcp-server convention tags (`mcp`, `ai`). + +The [`server.json` `repository` object](https://github.com/modelcontextprotocol/registry/blob/main/docs/reference/server-json/draft/server.schema.json) supplies a repository-root `url` (browse and git clone), a hosting-service `source` identifier, and an optional `subfolder` that is a clean relative path from the repository root. It does not supply a git branch or ref. The mapping therefore MUST NOT invent a branch name such as `main`. The mapping SHALL compute the combined repository URL as follows: + +1. **Normalize the base URL (for combination only).** Let `base` be `repository.url` with any trailing `/` removed and with a trailing `.git` suffix removed when present. This normalization applies only to the combined Source Code / `backstage.io/source-location` URL; it SHALL NOT alter the dedicated `modelcontextprotocol.io/repository.url` annotation. +2. **Absent subfolder.** When `repository.subfolder` is unset or empty after trimming whitespace, the combined URL is `base`. No tree/src/browse segment is inserted. +3. **Normalize the subfolder.** Let `subfolder` be `repository.subfolder` with leading and trailing `/` characters removed. The mapping SHALL treat it as `/`-separated relative path segments and SHALL NOT resolve `.` or `..` segments. +4. **Select a browse-path template from `repository.source`** (compared case-insensitively) so subdirectory URLs remain valid across SCMs. Templates that require a ref SHALL use the git symbolic ref `HEAD` (the repository default branch): + - `github` → `{base}/tree/HEAD/{subfolder}` + - `gitlab` → `{base}/-/tree/HEAD/{subfolder}` + - `bitbucket` → `{base}/src/HEAD/{subfolder}` + - `azure-devops` → `{base}` with query parameter `path=/{subfolder}` (use `?` when `base` has no query string, otherwise `&`; do not invent a `version`/`GB*` ref) + - any other or unset `source` → `{base}/{subfolder}` (path join). This fallback MAY not be a clickable browse URL on the hosting platform; `repository.source` and `repository.subfolder` remain projected as `modelcontextprotocol.io/*` annotations so a consumer can reconstruct a host-specific URL. +5. **Emit (when the original `repository.url` passes the emitted-URL scheme policy).** The `Source Code` `metadata.links` entry `url` is the combined URL. The `backstage.io/source-location` value is `url:` concatenated with the combined URL. The dedicated `modelcontextprotocol.io/repository.url` annotation is the original `repository.url` scalar, unnormalized. When the policy fails, none of those three fields are emitted. + +`repository.url` is treated as the repository root, matching the MCP schema; the mapping SHALL NOT parse or strip an existing `/tree//` (or equivalent) prefix from it. + +#### Scenario: Descriptive fields lift to metadata + +- **WHEN** a `server.json` provides `title`, `description`, and `websiteUrl` +- **THEN** the entity has `metadata.title` from `title`, `metadata.description` from `description`, and a `metadata.links` entry whose `url` is `websiteUrl` and whose `title` is `Website` + +#### Scenario: GitHub repository subfolder uses the GitHub tree path with HEAD + +- **WHEN** a `server.json` provides `repository.url` `https://github.com/modelcontextprotocol/servers`, `repository.source` `github`, and `repository.subfolder` `src/everything` +- **THEN** the entity has a `metadata.links` entry whose `url` is `https://github.com/modelcontextprotocol/servers/tree/HEAD/src/everything` and whose `title` is `Source Code`, a `backstage.io/source-location` annotation `url:https://github.com/modelcontextprotocol/servers/tree/HEAD/src/everything`, a dedicated `modelcontextprotocol.io/repository.url` annotation equal to the original `https://github.com/modelcontextprotocol/servers`, and the remaining `repository` sub-fields (`source`, `id`, `subfolder`) projected into `modelcontextprotocol.io/*` annotations + +#### Scenario: GitLab repository subfolder uses the GitLab tree path with HEAD + +- **WHEN** a `server.json` provides `repository.url` `https://gitlab.com/group/repo`, `repository.source` `gitlab`, and `repository.subfolder` `servers/weather` +- **THEN** the `Source Code` link URL and the `backstage.io/source-location` target (the portion after `url:`) are both `https://gitlab.com/group/repo/-/tree/HEAD/servers/weather` + +#### Scenario: Bitbucket repository subfolder uses the Bitbucket src path with HEAD + +- **WHEN** a `server.json` provides `repository.url` `https://bitbucket.org/org/repo`, `repository.source` `bitbucket`, and `repository.subfolder` `src/server` +- **THEN** the `Source Code` link URL and the `backstage.io/source-location` target are both `https://bitbucket.org/org/repo/src/HEAD/src/server` + +#### Scenario: Azure DevOps repository subfolder uses a path query parameter + +- **WHEN** a `server.json` provides `repository.url` `https://dev.azure.com/org/project/_git/repo`, `repository.source` `azure-devops`, and `repository.subfolder` `src/server` +- **THEN** the `Source Code` link URL and the `backstage.io/source-location` target are both `https://dev.azure.com/org/project/_git/repo?path=/src/server` + +#### Scenario: Unknown SCM falls back to path join + +- **WHEN** a `server.json` provides `repository.url` `https://git.example.com/org/repo`, `repository.source` `gerrit`, and `repository.subfolder` `src/server` +- **THEN** the `Source Code` link URL and the `backstage.io/source-location` target are both `https://git.example.com/org/repo/src/server` + +#### Scenario: Repository without subfolder uses the base URL + +- **WHEN** a `server.json` provides `repository.url` `https://github.com/org/repo` and no `repository.subfolder` +- **THEN** the `Source Code` link URL and the `backstage.io/source-location` target are both `https://github.com/org/repo`, with no `/tree/HEAD` segment inserted, and `modelcontextprotocol.io/repository.url` is `https://github.com/org/repo` + +#### Scenario: Original repository.url is preserved unnormalized + +- **WHEN** a `server.json` provides `repository.url` `https://github.com/org/repo.git/` (trailing `.git` and `/`) +- **THEN** `modelcontextprotocol.io/repository.url` is exactly `https://github.com/org/repo.git/`, while the `Source Code` link URL and the `backstage.io/source-location` target use the normalized base `https://github.com/org/repo` + +#### Scenario: javascript: websiteUrl is not copied into metadata.links or annotations + +- **WHEN** a `server.json` provides `websiteUrl` `javascript:alert(1)` +- **THEN** no `metadata.links` entry titled `Website` is emitted, the mapping still succeeds, and `websiteUrl` is not projected into a `modelcontextprotocol.io/*` annotation + +#### Scenario: data: repository.url is not copied into URL-shaped fields + +- **WHEN** a `server.json` provides `repository.url` `data:text/html,` +- **THEN** no Source Code `metadata.links` entry, no `backstage.io/source-location` annotation, and no `modelcontextprotocol.io/repository.url` annotation are emitted + +#### Scenario: https repository URL with an internal-looking hostname is still copied + +- **WHEN** a `server.json` provides `repository.url` `https://gitlab.internal/org/repo` and no `repository.subfolder` +- **THEN** the Source Code link URL, the `backstage.io/source-location` target, and `modelcontextprotocol.io/repository.url` are all `https://gitlab.internal/org/repo`; the mapping copies them because the scheme is `https`, not because it classified the host as internal + +#### Scenario: mcp-server tags applied + +- **WHEN** any `server.json` is mapped +- **THEN** `metadata.tags` includes `mcp` and `ai` + +### Requirement: Emitted URL scheme policy + +Before copying a source URL into an **emitted URL field** or projecting it as a `modelcontextprotocol.io/*` annotation, the mapping SHALL gate the candidate on scheme. Emitted URL fields are: `metadata.links[].url`, the target of `backstage.io/source-location` (the substring after `url:`), `spec.remotes[].url`, and `modelcontextprotocol.io/repository.url`. Projection of URL-typed leaves (`websiteUrl`, `repository.url`, `remotes[].url`, `icons[].src`, and any other scalar that WHATWG-parses as a URL whose protocol is not `http:` or `https:`) SHALL use the same gate. + +The mapping SHALL trim leading and trailing whitespace, then parse the candidate as an **absolute** URL **without a base URL** (relative paths, scheme-relative `//host`, and scp-like `git@host:path` SHALL fail). The candidate passes only when parse succeeds and the scheme is `http` or `https` (case-insensitive; equivalent to a WHATWG `URL` whose `protocol` is `http:` or `https:`). Schemes such as `javascript:`, `data:`, `file:`, `vbscript:`, and `blob:` SHALL NOT pass. The mapping SHALL NOT fail the transform solely because a candidate was refused. A refused URL SHALL NOT be copied into an emitted URL field and SHALL NOT be projected under any annotation key. Non-URL scalars whose absolute-URL parse fails (for example package identifiers) SHALL still project. + +Host appearance SHALL NOT affect the scheme gate. Candidates such as `http://localhost:7007/api/mcp/v1`, `http://10.0.0.5:8080/mcp` (a private-looking IPv4 literal; not a guarantee of operator-internality), and `https://gitlab.internal/org/repo` SHALL pass if and only if they are absolute `http`/`https`, the same as any other host. The mapping SHALL NOT classify hosts as public vs private, SHALL NOT DNS-resolve, fetch, or otherwise dereference any URL, and SHALL NOT omit an emitted URL field because a host looks internal. Trust of registry-supplied URLs is an ingestion-operator concern (the registry is a trusted input to this transform), not a mapping concern. Docs MAY use a TEST-NET address (RFC 5737, e.g. `192.0.2.1`) as a reserved example that will not collide with a real VPC; TEST-NET is not RFC 1918. + +#### Scenario: javascript: and data: schemes are refused for emitted URL fields and projection + +- **WHEN** a candidate URL uses scheme `javascript:` or `data:` (any casing, including with surrounding whitespace) +- **THEN** that candidate is not copied into `metadata.links`, `backstage.io/source-location`, `spec.remotes[].url`, `modelcontextprotocol.io/repository.url`, or any other `modelcontextprotocol.io/*` annotation + +#### Scenario: Non-absolute and scp-like URLs are refused for URL-typed leaves + +- **WHEN** a URL-typed leaf (`websiteUrl`, `repository.url`, `remotes[].url`, `icons[].src`) is `/relative/path`, `//evil.example/path`, or `git@github.com:org/repo.git` +- **THEN** that candidate is not copied into an emitted URL field and is not projected as an annotation; remaining non-URL siblings still project + +#### Scenario: http(s) URLs with private-looking hosts pass the same scheme gate + +- **WHEN** a candidate is `http://localhost:7007/api/mcp/v1`, `http://10.0.0.5:8080/mcp`, or `https://gitlab.internal/org/repo` +- **THEN** the candidate passes the scheme policy because it is absolute `http`/`https`, and is eligible to be copied into the corresponding emitted URL field + +### Requirement: Supply catalog-required fields absent from server.json + +`server.json` does not carry a Backstage owner or lifecycle. The mapping SHALL set `spec.owner` to the constant `unknown` when no caller override is supplied, and to a caller-provided default when one is supplied; the mapping SHALL NOT fail for a missing owner. The mapping SHALL set `spec.lifecycle` to the constant `production` when no caller override is supplied, and to a caller-provided default lifecycle when one is supplied; the mapping SHALL NOT fail for a missing lifecycle. + +#### Scenario: Owner defaults to unknown + +- **WHEN** the mapping is invoked without a caller-provided owner and `server.json` carries no owner information +- **THEN** `spec.owner` is set to `unknown` and the mapping succeeds + +#### Scenario: Owner override supplied by caller + +- **WHEN** the mapping is invoked with a caller-provided default owner +- **THEN** `spec.owner` is set to that owner value + +#### Scenario: Lifecycle defaults to production + +- **WHEN** the mapping is invoked without a caller-provided lifecycle +- **THEN** `spec.lifecycle` is set to `production` and the mapping succeeds + +#### Scenario: Lifecycle override supplied by caller + +- **WHEN** the mapping is invoked with a caller-provided default lifecycle +- **THEN** `spec.lifecycle` is set to that lifecycle value + +### Requirement: Deterministic and idempotent mapping + +The mapping SHALL be a pure function of its inputs (the `server.json` document and the caller-provided defaults): given identical inputs it SHALL produce a byte-identical entity, with stable ordering of `spec.remotes`, `metadata.tags`, and annotation keys, and SHALL NOT introduce timestamps, random values, or ingestion-source state. + +#### Scenario: Same input yields identical output + +- **WHEN** the mapping is run twice on identical inputs +- **THEN** the two produced entities are identical, including annotation key ordering and `spec.remotes` ordering diff --git a/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/tasks.md b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/tasks.md new file mode 100644 index 00000000000..1b4b889b017 --- /dev/null +++ b/workspaces/ai-integrations/openspec/changes/mcp-registry-server-mapping/tasks.md @@ -0,0 +1,42 @@ +# Tasks: MCP Registry Server Mapping + + + + +## 1. Mapping Reference & Schema Pinning + +- [ ] 1.1 Pin the source `server.json` draft schema version this mapping targets and record it (with URL and retrieval date) in a `mapping-reference.md` under the change +- [ ] 1.2 Author the canonical field-mapping table in `mapping-reference.md`: each `server.json` attribute → native entity target OR "projected annotation", including `name`→`metadata.name`(`____`)+`modelcontextprotocol.io/name`, `version`→`modelcontextprotocol.io/version`, `description`/`title`, `websiteUrl`→`metadata.links` (title `Website`) when the URL passes D11, `repository.url`→ dedicated `modelcontextprotocol.io/repository.url` (original scalar, unnormalized) **only when D11 passes** **and** (`+ repository.subfolder` via the SCM-aware combination algorithm keyed on `repository.source`)→`metadata.links` (title `Source Code`) + `backstage.io/source-location` (value `url:`) when `repository.url` passes D11, emitted convention tags `mcp`/`ai` (not a `server.json` field), `remotes[]`→`spec.remotes[]` when each `url` passes D11 +- [ ] 1.3 Document the annotation key rules (dot-separated `modelcontextprotocol.io/attribute.tree.to.leaf`, character sanitization, 63-char truncation + stable hash suffix, collision disambiguation) with worked examples +- [ ] 1.4 Document field-supply rules (`owner` defaults to the constant `unknown`, `lifecycle` defaults to the constant `production`, identity `prefix` defaults to the constant `mcp.registry` — all overridable by caller defaults, never a failure; empty prefix falls back to `mcp.registry`) and the null/empty-container omission rule +- [ ] 1.5 Document the emitted-URL scheme policy (D11): allowlist absolute `http`/`https` only on `metadata.links[].url`, `backstage.io/source-location`, `spec.remotes[].url`, `modelcontextprotocol.io/repository.url`, and every projected URL annotation (`websiteUrl`, `remotes[].url`, `icons[].src`, …); refuse `javascript:`/`data:`/`file:`/`vbscript:`/`blob:`/relative/`//host`/scp-like `git@host:path`; do not classify hosts (fixtures `localhost`, `10.0.0.5`, `gitlab.internal` pass because they are `http`/`https`); no fetch/DNS; refused URLs are omitted from native fields **and** not projected + +## 2. Direct Field Mapping Implementation + +- [ ] 2.1 Implement the `server.json` → `mcp-server` `API` entity transform skeleton (pure function of the document plus caller defaults; no I/O, no timestamps, no randomness) +- [ ] 2.2 Implement identity derivation: sanitize and combine `____` for `metadata.name` (prefix defaults to `mcp.registry`, caller override allowed, empty or sanitizes-to-empty falls back to the default), preserve bare name in `modelcontextprotocol.io/name`, version in `modelcontextprotocol.io/version`, and append a hash suffix derived from prefix, canonical name, and version whenever sanitization mutates any identity segment or the candidate is truncated to ≤63 characters +- [ ] 2.3 Implement descriptive-metadata mapping (`title`, `description`, `websiteUrl`→`metadata.links` entry titled `Website` only when D11 passes, original `repository.url` copied verbatim to `modelcontextprotocol.io/repository.url` only when D11 passes, `repository.url` combined with `repository.subfolder` when present via the SCM-aware algorithm in `mcp-registry-server-mapping` — `github`/`gitlab`/`bitbucket` HEAD browse-path templates, `azure-devops` `path=/{subfolder}` (no HEAD), path-join fallback →`metadata.links` entry titled `Source Code` **and** a `backstage.io/source-location` annotation whose value is `url:` only when D11 passes, emitted convention tags `mcp`/`ai`) +- [ ] 2.4 Implement `remotes[]` → top-level `spec.remotes[]` (`type`, `url`, source order preserved for remotes that pass D11); omit a remote whose `url` fails D11 (project its `type` and non-URL sub-fields, **not** the refused `url`); when the source has no remotes (empty or unset) or every remote is refused, emit an empty `spec.remotes: []` array (never omitted); ensure no `spec.definition` is emitted +- [ ] 2.5 Implement field-supply: `spec.type: mcp-server`, `spec.owner` set to `unknown` by default (caller override allowed, never a failure), `spec.lifecycle` set to `production` by default (caller override allowed, never a failure) +- [ ] 2.6 Enforce required-source-field validation (`name`, `description`, `version`) with actionable errors that name the missing field +- [ ] 2.7 Implement D11 emitted-URL gating (trim; absolute parse with no base; allowlist `http`/`https` only on links, source-location, remotes, `modelcontextprotocol.io/repository.url`, and projected URL annotations; refuse `javascript:`/`data:`/`file:`/`vbscript:`/`blob:`/relative/`//host`/scp-like `git@host:path`; do not classify hosts; never fetch or DNS-resolve) + +## 3. Annotation Projection Implementation + +- [ ] 3.1 Implement the recursive scalar-leaf walker over `server.json` that builds dot-separated paths (object keys, zero-based array indices) +- [ ] 3.2 Implement key sanitization (illegal chars incl. `/`, `$`, `@`, whitespace, leading `_`), the ≤63-char truncation + stable hash suffix derived from the full source path, and sanitization-collision disambiguation +- [ ] 3.3 Implement value serialization (scalars to strings) and the null/empty-container omission rule +- [ ] 3.4 Implement the no-overwrite guard so generic projection never clobbers native/dedicated/reserved annotations (`modelcontextprotocol.io/name`, `modelcontextprotocol.io/version`, `modelcontextprotocol.io/repository.url`, `backstage.io/source-location`, …) and skips source fields only when they were actually consumed by a native mapping; do **not** project any D11-refused URL under any key (known fields `websiteUrl` / `*.url` / `*.src`, and any scalar that WHATWG-parses as a URL whose protocol is not `http:` or `https:`); still project non-URL siblings (`remotes[].type`, `icons[].mimeType`) +- [ ] 3.5 Wire remote sub-fields (`headers`, `variables`) and non-native sections (`packages`, `icons`, `repository.source`/`id`/`subfolder`, `_meta`) through projection so nothing is dropped except D9 secrets and D11-refused URLs (including `icons[].src` that fail D11) +- [ ] 3.6 Implement secret redaction (D9): when an `Input` object declares `isSecret: true`, prune its `default`/`value`/`choices` leaves from projection (uniformly across `environmentVariables`, remote `headers`/`variables`, and arguments) while still projecting the input's non-secret leaves (`name`, `description`, `format`, `isRequired`, `isSecret`, `placeholder`) + +## 4. Conformance Fixtures, Verification & Docs + +- [ ] 4.1 Create input→expected-output fixtures: minimal server (default prefix `mcp.registry`), catalog-valid short name with no hash, reverse-DNS name with hash (sanitization changed the identity), multi-version (same name, two versions → distinct hashed entities), caller prefix override, empty prefix fallback, server with `packages`/`icons`/`_meta`, server with remote `headers`/`variables`, over-length truncation+hash name cases, no-remotes and all-remotes-D11-refused cases expecting `spec.remotes: []`, and repository URL combination cases (`github`/`gitlab`/`bitbucket`/`azure-devops` subfolder, unknown-source path-join fallback, url-only with no subfolder, and unnormalized `repository.url` with trailing `.git`/`/` preserved verbatim on `modelcontextprotocol.io/repository.url`) +- [ ] 4.2 Verify every produced entity passes the upstream `mcp-server` `API` entity schema (`McpServerApiEntity`, PR #34016) — `spec.remotes` required, `spec.definition` not required — rather than the generic base `API` schema +- [ ] 4.3 Verify determinism/idempotency (byte-identical output on repeated runs) and scalar round-trip fidelity (every non-null, non-redacted source scalar recoverable — `isSecret: true` `default`/`value`/`choices` leaves and D11-refused URLs exempt) via tests over the fixtures +- [ ] 4.4 Add unit tests covering annotation key sanitization, truncation, and collision disambiguation edge cases +- [ ] 4.5 Write the user/consumer documentation: the mapping guide, annotation-key conventions, and a worked `server.json`→entity example (aligned with the upstream `backstage-mcp-server-api.yaml` shape) +- [ ] 4.6 Add a fixture and test covering secret redaction (D9): an `isSecret: true` env var / remote header with a populated `default`/`value` and `choices` produces no annotation carrying those values, while non-secret sibling leaves (`name`, `isSecret`, `placeholder`) still project; an `isSecret: false` input with `choices` still projects those choices +- [ ] 4.7 Add fixtures and tests for D11: `javascript:` and `data:` `websiteUrl`/`remotes[].url`/`icons[].src`/`repository.url` are omitted from emitted URL fields **and** from all projected annotations; `remotes[].type` and `icons[].mimeType` still project; `http://localhost` and `http://10.0.0.5` remotes plus `https://gitlab.internal` repository URLs are copied because they are `http`/`https` (not because the host is classified internal); the mapping does not fetch