Expand conformance coverage beyond the initial core plugin

[jonathanhefner]

Track Agent Plugins 1.0.0 coverage beyond the initial Core plugin. Close this issue once the remaining candidates have been implemented or explicitly deferred. Existing coverage is documented in the suite README and the linked fixture READMEs.

Completed

• Plugin data writability. Check plugin data writability through native observations #4

• A valid skill and a valid stdio MCP server remain available alongside combined recoverable manifest, skill, and server-entry errors. Add combined recovery fixture coverage #6

• Consistent data-directory assignment among Core servers and distinct data directories between Core and Recovery. Check plugin data directory assignments #7

• Valid immediate-child skill availability alongside additional directory layouts. Exercise core skill discovery with additional directory layouts #8

• Valid skill availability alongside a malformed whole MCP document. Check skill availability with a malformed MCP document #9

• Optional Streamable HTTP availability and literal URL, query, and header delivery. Add optional Streamable HTTP conformance checks #10

• Malformed YAML skill frontmatter in the valid-neighbor recovery check. Exercise recovery from malformed skill frontmatter #12

• Client-generated protocol header precedence over a conflicting configured Accept header. Exercise HTTP availability with a conflicting Accept header #13

• Exclusion of the nested reference skill from discovery. Verify nested skill exclusion in discovery checks #16

• Configured headers are not forwarded to another origin without authorization during an HTTP redirect. Check configured headers on cross-origin HTTP redirects #17

• Exclusion of an MCP server whose plugin-relative cwd escapes the plugin root. Check MCP server exclusion for an escaping relative cwd #18

• Exclusion of an MCP server whose ${PLUGIN_DATA}-rooted cwd escapes the plugin data directory. Check MCP server exclusion for an escaping data cwd #19

• Exclusion of an otherwise runnable MCP server with an unknown configuration field. Check MCP server exclusion for an unknown configuration field #20

• Exclusion of otherwise runnable HTTP servers whose URLs contain fragments or whose configured header names duplicate one another under different casing. Check HTTP server exclusion for fragments and duplicate headers #21

• Exclusion of otherwise runnable stdio MCP servers that configure reserved PLUGIN_ROOT or PLUGIN_DATA environment keys. Check MCP server exclusion for reserved environment variables #22

• Package-path symlink containment, exercised through an MCP server working directory; filesystem checks grouped independently of their observation mechanism. Check symlink containment and group filesystem requirements #23

• Exclusion of an otherwise runnable HTTP server whose URL contains user information. Check exclusion of HTTP URLs containing user information #24

• Exclusion of otherwise runnable HTTP servers with an invalid configured header name or value. Check exclusion of invalid HTTP header names and values #25

• Exclusion of a stdio MCP server whose cwd uses an invalid path form, independently of containment. Check exclusion of invalid working-directory syntax #26

• Exclusion of an HTTP server with a scheme-relative URL (//host/path). Check exclusion of relative HTTP URLs #27

• Exclusion of an otherwise runnable MCP entry missing its explicit transport type, guarding against legacy-format transport inference. Check exclusion of MCP entries missing a transport type #28

• Optional legacy HTTP+SSE availability and literal initial-connection URL and header delivery, with unsupported-transport diagnostics preserved. Add optional legacy HTTP+SSE conformance checks #29

• Legacy HTTP+SSE parity: configured headers on message requests, client-generated header precedence, cross-origin redirect and endpoint-event protection, and the six invalid URL/header entry checks. Bring legacy SSE checks to parity with Streamable HTTP #30

• Exclusion of an otherwise runnable MCP entry using the legacy http transport type, guarding against accepting a legacy-format alias for streamable-http. Check exclusion of the legacy HTTP transport type #31

• Preservation of a stdio command path as one executable token, with attributable evidence when the command is split. Check stdio command token preservation in Core #32

• Exclusion of non-loopback plaintext HTTP URLs for Streamable HTTP and legacy HTTP+SSE. Check rejection of non-loopback plaintext HTTP URLs #33

• Exclusion of an MCP server whose command escapes the plugin root through an executable symlink, with POSIX and Windows fixtures. Check MCP command symlink containment #34

• Plugin-relative commands resolve against the plugin root independently of the configured working directory, with attributable wrong-directory evidence and independent command-token verdicts. Check plugin-relative command resolution #36

• Strengthened the executable-symlink containment fixture with an in-package first hop followed by an external launcher, preserving exclusion evidence when either hop is removed. Exercise two-hop executable symlink containment #37

Remaining

• Assessed remaining malformed-package and invalid-configuration rejection: unsupported or unrecognized MCP schema identifiers are deferred below, alongside fatal manifest rejection and cross-version mismatch.
• Assessed missing and wrong-kind fixed component locations: dedicated checks are deferred below; do not count incidental loading of the guide or malformed-MCP recovery as coverage of these exact requirements.
• Assessed remaining filesystem containment: MCP working-directory and executable-symlink checks include a two-hop command chain (Exercise two-hop executable symlink containment #37). Manifest, fixed-location and skill paths, generic package-file access, and Windows junctions or other reparse mechanisms are deferred below.
• Configured environment values replace conflicting base-environment values, using naturally supplied USER or USERNAME as the control without additional client setup. Check configured environment precedence #35
• Assessed remaining invalid HTTP URL and header forms: retain representative coverage from Check HTTP server exclusion for fragments and duplicate headers #21, Check exclusion of HTTP URLs containing user information #24, Check exclusion of invalid HTTP header names and values #25, Check exclusion of relative HTTP URLs #27, Bring legacy SSE checks to parity with Streamable HTTP #30, and Check rejection of non-loopback plaintext HTTP URLs #33. Additional malformed forms need an independent rationale; non-HTTP(S) schemes are explicitly deferred below.

Each proposed check should identify its specification requirement, the evidence needed to judge it, and any additional setup or plugin installation it requires. Continued availability of valid components does not by itself establish rejection of invalid components. Missing evidence remains not_verified; client developers determine which results they require to pass.

Considered and skipped

• Exclusion of the malformed Recovery skill: deliberately retain valid-neighbor availability as the measured behavior. This preserves the earlier scope decision about enforcing invalid-skill rejection, reaffirmed after considering an explicit catalog-based exclusion check.

• Inline MCP configuration in plugin.json: the explicit prohibition records the removal of earlier configurable/inline discovery machinery to reduce implementation burden. Its wording does not establish a distinct implementation risk sufficient to justify this fixture.

Deferred

• Remaining filesystem containment: manifest, fixed component locations, and discovered skill files need valid external targets that remain available after installation, without assumptions about neighboring layouts or extra preparation. Generic package files lack a standardized client access operation for a fixture; subprocess access does not test client containment. Windows file symlinks are covered, but no portable fixture without additional setup has been established for junctions or other reparse mechanisms. Revisit with an attributable client operation, portable target, or concrete implementation defect.

• Unsupported or unrecognized MCP schema identifiers: untested. The malformed-MCP fixture checks recovery from invalid JSON, not schema selection. A dedicated plugin or installation state is not currently justified while 1.0.0 is the only published specification version and no concrete compatibility problem has been established. Revisit when another version is published or evidence of incorrect client behavior appears.

• Dedicated checks for missing or wrong-kind fixed component locations: the guide has no mcp.json, but successful loading does not establish that no erroneous diagnostic was emitted. Wrong-kind mcp.json and skills locations are distinct untested inputs; retaining current fixtures would require additional package states or installations. The existing malformed-MCP fixture exercises the related recovery behavior, and the incremental coverage does not currently justify more installations. Revisit if concrete client evidence raises their priority.

• Absolute MCP endpoint URLs using schemes other than HTTP or HTTPS: untested. No concrete compatibility risk or useful fixture distinguishing incorrect acceptance from ordinary connection failure has been established. Revisit with such evidence; the current URL/header checks provide representative rather than exhaustive coverage.

• Cross-version mismatch between plugin.json and mcp.json: requires a dedicated fixture plugin, and 1.0.0 is currently the only published specification version (1.1.0 is a working draft). Revisit when another version is published and the additional fixture is justified.

• Plugin data persistence across updates: update orchestration and evidence transfer across lifecycle phases are outside the work being pursued at this time.

• Fatal invalid-manifest rejection: correct rejection can happen during installation, before the guide can collect its diagnostic.

• Server startup-failure isolation: deferred until attempted startup can be reliably observed.

• Non-recursive expansion with placeholder-containing installation paths: set aside after assessing its practical value and setup cost.

Native-client execution of the existing suite was tracked separately in #3. PR #38 consolidates these limits and their rationale into documentation and closes this issue on merge. The completed checklist remains here as the coverage history.