Show what each host comparison checked, partially checked and could not establish

[pengfei-threemoonslab]

Problem

A reviewer receiving zero capability rows cannot tell whether the relevant change was compared, unsupported, or missed by discovery. Requiring separate base/head audits to reconstruct that answer makes the result hard to trust and reuse.

The September 15 exploratory 1.0.0 run recorded comparable, zero-row results for plugin MCP files, hook packages and marketplace references that the host entry did not read (feedback F4/F8). Those are dated entry-specific observations, not a claim that every Agents Shipgate reader lacks support. The reported 13/120 file ratio also included scripts and documentation; it is not a product-wide coverage rate.

User job: understand exactly which part of this PR the result establishes, including when no changes were found.

Proposed behavior

Add a compact coverage explanation to the existing comparison result and linked detail. Prioritize relevant changed inputs and their supported references; keep the full inventory in the detailed artifact.

For each item, show:

• Repository-relative source, comparison side/ref and the relevant supported input role.
• Whether the declared surface was compared, only partially established, unsupported by this entry, or unresolved.
• The concrete limitation and a supported next step when one exists.
• The reference/selection basis when a manifest selects a custom file.

Example wording (illustrative, not new schema names):

Input	What this run established
.claude/settings.json	Compared supported permission fields; two changes
plugins/demo/mcp.json	Candidate input; unsupported by this entry
plugins/demo/hooks.json	Referenced input; selection could not be resolved

A file appearing in sources_observed proves that it was observed, not that all of its fields or runtime effects were understood. Names resembling MCP/hooks/plugin files are discovery hints only. Use a bounded, documented candidate rule set and observed references; ordinary documentation or an unrelated script must not create a capability finding or fail a gate.

The summary must disclose truncated or incomplete discovery. Absence from a bounded candidate list cannot establish whole-repository coverage. Hashes/ref identities support reproducibility; they do not replace the explanation.

Scope and integration

• Start with the existing manifest-free host diff, maintained JSON and the selected advisory PR/linked-report path from Show concrete field changes so an unfamiliar reviewer can act on the first advisory result #795/[Feature]: Make deterministic permission review tangible through a change–correction–recurrence loop #811.
• Derive data from the same inventory/reader evidence used by the comparison; do not create a second scanning or verdict engine.
• Reuse Publish the determinism boundary: a generated coverage report of what shipgate can prove, per source type and shape #473's generated support metadata where applicable. Publish the determinism boundary: a generated coverage report of what shipgate can prove, per source type and shape #473 describes general supported shapes; this issue explains the actual inputs of one run.
• Distinguish discovered Claude hook artifacts from proven host loading #714 owns discovered-versus-selected plugin provenance. Report its facts when available without waiting for every plugin format to be supported.
• Preserve independently established host changes when a plugin scope is incomparable #808 owns retaining independent rows when a scope is incomparable. This issue explains coverage; it does not relax comparison/refusal rules.
• First delivery adds no remote fetch, plugin installation, runtime execution, new manifest or human declaration. Broader plugin readers remain separately scoped.

Acceptance

• The real entry distinguishes covered no-change, unsupported relevant input, partially read source, unresolved reference and complete comparison with changes.
• A supported file with an unread field/shape is not described as completely understood merely because it was observed.
• Base-only/deleted sources and manifest-selected custom filenames retain attributable coverage evidence.
• Candidate discovery is bounded and reports incompleteness/truncation; filename matches never assert active capability or runtime loading.
• CLI/JSON/PR detail agree on the same refs and coverage. A reviewer does not need two extra audit commands to explain a zero-row result.
• Existing public enums/consumers stay compatible under STABILITY.md; any necessary additive fields have consumer tests. Coverage hints do not manufacture a verdict or authority.
• Sensitive paths/values remain redacted without corrupting internal identity, and summarized output reports omitted items.
• [Feature]: Make deterministic permission review tangible through a change–correction–recurrence loop #811's known-change/correction/recurrence example includes a coverage explanation that a reviewer can identify unaided; actual observations stay in Design-partner conversations: two waves of five and at least three real host diffs #653/Run the published advisory host-diff pilot: first reviewer value and the next eligible change #571.

Priority and measures

P1, immediately after concrete before/after presentation in #795. Measure whether a reviewer correctly distinguishes “no supported change found” from “this input was not established”; do not optimize raw source counts or unsupported-file warning volume.

Related #811, #795, #714, #808, #473, #590. Static-by-default throughout.

Journey delivery - 2026-09-16

Program: #778. Delivery milestone: Adoption Phase 1 - A real change the reviewer understands. Accountable role: Reader/comparison and report engineering.

Explain what this particular base/head run established, next to #795's concrete changes, including a covered zero-row result. Distinguish supported compared fields, partial interpretation, unsupported relevant candidates and unresolved references; discovery is not proof of runtime loading. Reuse existing evidence and bounded discovery, with truncation visible. Do not wait for broader adapters or #808 to ship the first explanation. #808 later retains independently proven rows without weakening the gate. User acceptance is correct interpretation of no-change versus not-established, observed in #653/#571, not a larger inventory count.

Planning checkpoints preserve #778's original 2026-09-14 start: Oct 14 / Nov 13 / Dec 13. They are evidence-review dates, not promises about external replies. Phases overlap; a later milestone never delays supported research, an already-possible correction or an individual four-week opportunity window. A named accountable owner is required before execution; the role here is not an assignment to an uninvolved person.

[pengfei-threemoonslab]

Split — 2026-09-16 (owner decision, see #778)

Slice 1 stays here (Phase 1). It uses only facts the comparator already computes:

• Blocking sources and kinds behind an incomparable result. The refusal itself does not change.
• Compared sources with changes ("compared; N changes").
• Observed artifact changes with no grant change ("observed a change in fields this entry does not read").
• Base/head sources_observed deltas, so deleted or base-only sources stay attributable.
• Unresolved plugin references and unselected plugin hook candidates, worded as "candidate, not read, loading not established".

Slice 2 moves to #821 (Phase 2): bounded candidate discovery and named limits for relevant inputs the host entry does not read.

Fixture groups (synthetic, main 9df307ab)

1. Incomparable, no source named (slice 1). In the two link cases the head adds Bash(*) and drops a deny, and the control without a link gives 2 rows. In the SKILL.md case the head only drops a deny.

Fixture	Result
In-tree directory link outside boundary paths, docs/proxy -> ../src/pkg	incomparable, 0 rows
Unchanged dangling link NOTES.md -> does/not/exist.md	incomparable, 0 rows
Unchanged SKILL.md with non-string metadata, reached through .claude/skills -> ../.agents/skills (see #822)	incomparable, 0 rows

diff text prints Cannot compare against HEAD~1: base_inventory_incomplete; head_inventory_incomplete and a generic input-limit note, naming no source. The diff --json keys (base_commit, base_ref, capability_diff_schema_version, comparison_status, incomparable_reasons, rows, static_analysis_only, unchanged_limits, workspace) name no source. audit --host --json already has it, for example {kind: unreadable, source: docs/proxy, blocking: true}, so no new engine is needed.

2. Observed change, no grant change (slice 1). Adding outputStyle and env to .claude/settings.json gives comparable, 0 rows, "No static host-grant changes detected."

3. Comparable zero-row results where relevant candidates were not read (#821). Each of these gives comparable, 0 rows, 0 audit artifacts and host coverage complete:

• Cursor plugin mcp.json
• root mcp.json
• inline plugin mcpServers
• Codex plugin hooks
• .cursor/hooks.json guard removal
• a claude/.claude/settings.json dotfiles package
• an external marketplace sha change

4. Parity gap.

• verify --preview exposes host_comparison.paths only when a host artifact is readable. For the Cursor plugin mcp.json fixture, host_comparison is null.
• With .claude/settings.json and .cursor/hooks.json both changed, paths is [".claude/settings.json"]: the changed .cursor/hooks.json is neither listed nor flagged.
• Standalone diff has no equivalent field.

Scan context (2026-09-15, 23 public PRs, aggregated): 9 of 18 comparable results were silent zero-row results, and 0 of 5 incomparable results named a source. By its definition, slice 1 addresses the 5 incomparable results and group 2. #821 addresses group 3.