Add skill distribution audit with discovery reconciliation

docs/README.skills.md

@@ -418,6 +418,7 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to
 | [shopify-review-triage](../skills/shopify-review-triage/SKILL.md)<br />`gh skills install github/awesome-copilot shopify-review-triage` | Use this skill when someone wants public Shopify App Store reviews, low-star reviews, or merchant feedback triaged, prioritized, clustered, or turned into a product or support brief. Trigger for prompts like "triage these app store reviews", "what should we fix first from this feedback", "cluster our 1-star reviews", or "write a weekly low-star review brief", for a single Shopify app or a portfolio plus watched competitors. Produces a P0-P3 brief covering incident risk, repeated friction, pricing confusion, feature requests, and an explicit needs-human-read bucket, where every item keeps its public source link and stays labeled first pass or human-checked. Do not trigger for support tickets, order data, or any other private merchant data, and never use it to reply to or contact a reviewer. | None |
 | [shuffle-json-data](../skills/shuffle-json-data/SKILL.md)<br />`gh skills install github/awesome-copilot shuffle-json-data` | Shuffle repetitive JSON objects safely by validating schema consistency before randomising entries. | None |
 | [signal-write](../skills/signal-write/SKILL.md)<br />`gh skills install github/awesome-copilot signal-write` | Emit structured agent signals — hands-up, blocked, done, checkpoint, partnership. Signals are written as JSON to .signals/ for dashboard consumption and noted in the journal for persistence. | None |
+| [skill-distribution-audit](../skills/skill-distribution-audit/SKILL.md)<br />`gh skills install github/awesome-copilot skill-distribution-audit` | Audit a skill collection before release or investigate installer omissions, archived/template leakage, duplicate identities, and manifest drift. Reconcile the intended public source set, distribution manifest, and actual discovery output for a specified consumer version. Includes a deterministic set-comparison helper. | `references/worked-example.md`<br />`scripts/compare-discovery.mjs`<br />`scripts/compare-discovery.test.mjs` |
 | [slang-shader-engineer](../skills/slang-shader-engineer/SKILL.md)<br />`gh skills install github/awesome-copilot slang-shader-engineer` | Use when working with Slang shaders, shader modules, HLSL-compatible GPU code, graphics pipelines, compute shaders, tessellation, ray tracing, parameter blocks, generics, interfaces, capabilities, cross-compilation, shader optimization, shader review, or C++ engine integration for Slang. Trigger on any mention of Slang, .slang files, slangc, SPIR-V from Slang, Slang modules, [shader("compute")], [shader("vertex")], or requests to write/review/refactor shader code with modern language features. Also trigger for Slang-to-HLSL/GLSL/Metal/CUDA cross-compile questions, or when the user says "shader" alongside "generics", "interfaces", "parameter blocks", "autodiff", or "capabilities". | `references/language-reference.md`<br />`references/rules-and-patterns.md`<br />`references/slang-documentation-full.md` |
 | [snowflake-semanticview](../skills/snowflake-semanticview/SKILL.md)<br />`gh skills install github/awesome-copilot snowflake-semanticview` | Create, alter, and validate Snowflake semantic views using Snowflake CLI (snow). Use when asked to build or troubleshoot semantic views/semantic layer definitions with CREATE/ALTER SEMANTIC VIEW, to validate semantic-view DDL against Snowflake via CLI, or to guide Snowflake CLI installation and connection setup. | None |
 | [speak-summary](../skills/speak-summary/SKILL.md)<br />`gh skills install github/awesome-copilot speak-summary` | Convert text, markdown, or a summary produced by another skill into a listenable MP3 using local CPU-only neural text-to-speech. Rewrites written prose for the ear before synthesising. Use when the user asks to "read this out", "turn this into audio", "make an MP3", "I want to listen to this", "podcast version", or wants a spoken digest for a commute or breakfast. | `scripts/tts.sh` |

skills/skill-distribution-audit/SKILL.md

@@ -0,0 +1,53 @@
+---
+name: skill-distribution-audit
+description: "Audit a skill collection before release or investigate installer omissions, archived/template leakage, duplicate identities, and manifest drift. Reconcile the intended public source set, distribution manifest, and actual discovery output for a specified consumer version. Includes a deterministic set-comparison helper."
+---
+
+# Skill Distribution Audit
+
+Give a collection maintainer a release-correctness report: which intended skills are missing, which unintended entries are exposed, and which identities are ambiguous. Start from the source revision and target consumer, rather than assuming that a readable SKILL.md will be distributed correctly.
+
+## Establish the distribution contract
+
+Record the source revision, consumer/client and version, distribution layout, and requested scope. If a client-specific rule affects the result, consult that consumer's documentation or implementation; otherwise leave the conclusion unverified.
+
+Build the intended public set from the repository's actual release policy and source files. Distinguish active skills from archives, templates, test fixtures, generated copies, and internal maintenance content. A source collection can contain SKILL.md files that it deliberately does not publish. An existing manifest may itself be stale, so do not use it as the only source of truth.
+
+Keep a mapping from each identity to its source path. Prefer repository-relative paths when both sides expose them; use names only when unique and comparable. If discovery returns names that map to multiple source paths, report the ambiguity rather than silently choosing or deduplicating an entry.
+
+## Collect comparable evidence
+
+- Parse frontmatter with a standard YAML parser compatible with the target consumer, then validate its supported field types. Separate malformed YAML from valid YAML that fails the consumer's schema. A repository's custom parser passing is not proof of consumer compatibility.
+- Inspect the actual release/manifest rules, including path bases, explicit entries, globs, and exclusion or internal flags. Apply only the target version's semantics: a repository's deprecated status field does not necessarily hide an entry from an installer.
+- Obtain the consumer's discovery/list output for the same source revision when an appropriate command is available and authorized. Inspect its behavior first; a command named list is not automatically free of downloads or writes. Prefer an isolated checkout and retain stderr, exit status, version, source revision, and completeness/pagination limits.
+- Use existing captured output when a fresh consumer run is unavailable. Mark it as historical. If no comparable consumer output is available, complete the static checks and state that discovery coverage remains unverified.
+
+Do not execute skill bodies or their helpers while collecting distribution evidence. Installation and end-to-end task execution are separate validation scopes; do not repeat installations to manufacture usage counts.
+
+## Reconcile identities, not just totals
+
+Compare the intended public set against the manifest and against observed discovery separately, when those layers exist. Report missing identities, unexpected identities, duplicates, and parse failures independently. Also compare the manifest to discovery when it is the actual consumer input. Do not assume that a manifest for a different installer controls a direct repository scan.
+
+Equal totals can conceal different identities. For example, an intended set of two active skills and an observed set containing one active skill plus one template both have length two, yet contain one omission and one leak.
+
+For deterministic comparisons, use the bundled [compare-discovery.mjs](scripts/compare-discovery.mjs) with Node.js 20 or later; it needs no third-party packages. It accepts two JSON arrays of exact identity strings. Create those inputs from the evidence above; it does not discover skills or infer release policy.
+
+```sh
+node <skill-directory>/scripts/compare-discovery.mjs <expected.json> <observed.json>
+```
+
+The JSON report includes raw/unique counts, missing and unexpected identities, and duplicate counts on both sides. Exit codes: 0 for an exact unique match, 1 for mismatches or duplicates, 2 for invalid input or an I/O error. It reads the supplied files and writes only its report; it never invokes an installer.
+
+See the [worked example](references/worked-example.md) for sample inputs and an auditable report. The helper's regression tests run with:
+
+```sh
+node --test <skill-directory>/scripts/compare-discovery.test.mjs
+```
+
+## Report and repair
+
+For each finding, give the affected identities, source/configuration paths and line numbers, observed behavior, likely cause, and smallest repair. Distinguish direct observations from inferences. Avoid changing skill bodies when a metadata or manifest correction is sufficient; preserve unrelated metadata and licensing.
+
+Include the source revision, consumer version, expected/manifest/discovered counts, exact differences, checks actually run, and verification limits. Allow zero findings. Do not turn unavailable or truncated discovery into a successful match by comparing only a convenient subset.
+
+An exact discovery match establishes distribution coverage within the audited scope. It does not establish that skills are loaded by a live Copilot session, that packaged helpers work, or that credentials and external services are usable. After an authorized repair, repeat the relevant parsing and reconciliation checks on the repaired revision and report what changed.

skills/skill-distribution-audit/references/worked-example.md

@@ -0,0 +1,38 @@
+# Equal counts can hide distribution failures
+
+This is a small synthetic example, not a benchmark of Copilot or an observation from a live installer.
+
+A collection's release policy publishes two skills, `active-a` and `active-b`, while keeping `template` internal. Its captured discovery output lists `active-a` and `template`. Both totals are two.
+
+Supply the intended identities as the first JSON input:
+
+```json
+["active-a", "active-b"]
+```
+
+Supply the observed identities as the second JSON input:
+
+```json
+["active-a", "template"]
+```
+
+Run [compare-discovery.mjs](../scripts/compare-discovery.mjs) as described in the skill. It returns exit code 1 with:
+
+```json
+{
+  "match": false,
+  "counts": {
+    "expected": 2,
+    "observed": 2,
+    "expectedUnique": 2,
+    "observedUnique": 2
+  },
+  "missing": ["active-b"],
+  "unexpected": ["template"],
+  "duplicates": {"expected": [], "observed": []}
+}
+```
+
+The report should then investigate each difference separately. For the omitted skill, inspect parser warnings, schema compatibility, path rules, and the target revision. For the template, inspect the release policy and the actual consumer's exclusion semantics. The identity comparison alone does not establish either root cause.
+
+If the manifest belongs to another distribution channel, compare that channel separately. Do not infer that changing that manifest will change direct repository discovery. After a repair, an exact unique match should return exit code 0; this still does not verify runtime loading or execution.

skills/skill-distribution-audit/scripts/compare-discovery.mjs

@@ -0,0 +1,61 @@
+#!/usr/bin/env node
+import fs from "node:fs";
+import { fileURLToPath } from "node:url";
+
+function countIdentities(values, label) {
+  if (!Array.isArray(values)) throw new TypeError(`${label} must be a JSON array`);
+  const counts = new Map();
+  for (const value of values) {
+    if (typeof value !== "string" || !value.trim() || value !== value.trim()) {
+      throw new TypeError(`${label} identities must be nonempty strings without surrounding whitespace`);
+    }
+    counts.set(value, (counts.get(value) ?? 0) + 1);
+  }
+  return counts;
+}
+
+export function compareDiscovery(expected, observed) {
+  const expectedCounts = countIdentities(expected, "expected");
+  const observedCounts = countIdentities(observed, "observed");
+  const missing = [...expectedCounts.keys()].filter(id => !observedCounts.has(id)).sort();
+  const unexpected = [...observedCounts.keys()].filter(id => !expectedCounts.has(id)).sort();
+  const duplicates = counts => [...counts.keys()].sort()
+    .filter(id => counts.get(id) > 1).map(id => ({ id, count: counts.get(id) }));
+  const expectedDuplicates = duplicates(expectedCounts);
+  const observedDuplicates = duplicates(observedCounts);
+  return {
+    match: !missing.length && !unexpected.length && !expectedDuplicates.length && !observedDuplicates.length,
+    counts: {
+      expected: expected.length, observed: observed.length,
+      expectedUnique: expectedCounts.size, observedUnique: observedCounts.size,
+    },
+    missing,
+    unexpected,
+    duplicates: { expected: expectedDuplicates, observed: observedDuplicates },
+  };
+}
+
+function isCliEntry() {
+  if (!process.argv[1]) return false;
+  try {
+    return fs.realpathSync(process.argv[1]) === fs.realpathSync(fileURLToPath(import.meta.url));
+  } catch {
+    return false;
+  }
+}
+
+if (isCliEntry()) {
+  try {
+    if (process.argv.length !== 4) {
+      throw new Error("Usage: node compare-discovery.mjs <expected.json> <observed.json>");
+    }
+    const expected = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
+    const observed = JSON.parse(fs.readFileSync(process.argv[3], "utf8"));
+    const report = compareDiscovery(expected, observed);
+    process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
+    process.exitCode = report.match ? 0 : 1;
+  } catch (error) {
+    process.stderr.write(`${error.message}\n`);
+    process.exitCode = 2;
+  }
+}

skills/skill-distribution-audit/scripts/compare-discovery.test.mjs

@@ -0,0 +1,113 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { spawnSync } from "node:child_process";
+import { compareDiscovery } from "./compare-discovery.mjs";
+
+test("an exact match is independent of input order and accepts empty sets", () => {
+  assert.equal(compareDiscovery(["b", "a"], ["a", "b"]).match, true);
+  assert.equal(compareDiscovery([], []).match, true);
+});
+
+test("equal totals still expose an omission and a leaked template", () => {
+  const result = compareDiscovery(["active-a", "active-b"], ["active-a", "template"]);
+  assert.equal(result.match, false);
+  assert.equal(result.counts.expected, result.counts.observed);
+  assert.deepEqual(result.missing, ["active-b"]);
+  assert.deepEqual(result.unexpected, ["template"]);
+});
+
+test("duplicates on either side invalidate otherwise equal sets", () => {
+  const result = compareDiscovery(["a", "a", "b"], ["b", "b", "a"]);
+  assert.deepEqual(result.missing, []);
+  assert.deepEqual(result.unexpected, []);
+  assert.equal(result.match, false);
+  assert.deepEqual(result.duplicates, {
+    expected: [{ id: "a", count: 2 }], observed: [{ id: "b", count: 2 }],
+  });
+});
+
+test("identities remain exact and input arrays remain unchanged", () => {
+  const expected = Object.freeze(["A", "group/a"]);
+  const observed = Object.freeze(["a", "group\\a"]);
+  const result = compareDiscovery(expected, observed);
+  assert.deepEqual(result.missing, ["A", "group/a"]);
+  assert.deepEqual(result.unexpected, ["a", "group\\a"]);
+});
+
+test("malformed identity collections are rejected instead of coerced", () => {
+  for (const value of [null, {}, "a", [""], [" a"], ["a "], [42], [null]]) {
+    assert.throws(() => compareDiscovery(value, []), TypeError);
+    assert.throws(() => compareDiscovery([], value), TypeError);
+  }
+});
+
+test("CLI reports exit codes, preserves inputs, and handles invalid JSON and missing files", t => {
+  const root = fs.mkdtempSync(path.join(os.tmpdir(), "skill-discovery-test-"));
+  // Remove only this test's newly created temporary directory.
+  t.after(() => {
+    assert.equal(path.dirname(path.resolve(root)), path.resolve(os.tmpdir()));
+    assert.ok(path.basename(root).startsWith("skill-discovery-test-"));
+    fs.rmSync(root, { recursive: true, force: true });
+  });
+  const script = fileURLToPath(new URL("./compare-discovery.mjs", import.meta.url));
+  const expected = path.join(root, "expected.json");
+  const observed = path.join(root, "observed.json");
+  fs.writeFileSync(expected, '["active-a","active-b"]\n');
+  const original = fs.readFileSync(expected);
+  const run = (...args) => spawnSync(process.execPath, [script, ...args], { encoding: "utf8", windowsHide: true });
+  for (const [input, status] of [
+    ['["active-b","active-a"]', 0],
+    ['["active-a","template"]', 1],
+    ['["active-a","active-b","active-b"]', 1],
+    ['{"a":true}', 2],
+    ["invalid JSON", 2],
+  ]) {
+    fs.writeFileSync(observed, input);
+    const result = run(expected, observed);
+    assert.equal(result.status, status, result.stderr);
+    assert.deepEqual(fs.readFileSync(expected), original);
+    assert.equal(fs.readFileSync(observed, "utf8"), input);
+    if (status < 2) assert.equal(JSON.parse(result.stdout).match, status === 0);
+    else assert.ok(result.stderr.length > 0);
+  }
+  assert.equal(run(expected, path.join(root, "missing.json")).status, 2);
+  assert.equal(run(expected).status, 2);
+  assert.deepEqual(fs.readdirSync(root).sort(), ["expected.json", "observed.json"]);
+});
+
+test("CLI works through a linked installation directory", t => {
+  const root = fs.mkdtempSync(path.join(os.tmpdir(), "skill-discovery-test-"));
+  const installed = path.join(root, "installed");
+  t.after(() => {
+    assert.equal(path.dirname(path.resolve(root)), path.resolve(os.tmpdir()));
+    assert.ok(path.basename(root).startsWith("skill-discovery-test-"));
+    if (fs.existsSync(installed)) {
+      assert.ok(fs.lstatSync(installed).isSymbolicLink());
+      fs.unlinkSync(installed);
+    }
+    fs.rmSync(root, { recursive: true, force: true });
+  });
+  const source = path.dirname(fileURLToPath(import.meta.url));
+  try {
+    fs.symlinkSync(source, installed, process.platform === "win32" ? "junction" : "dir");
+  } catch (error) {
+    if (!["EPERM", "EACCES", "ENOSYS"].includes(error.code)) throw error;
+    t.skip(`Links unavailable: ${error.code}`);
+    return;
+  }
+  const expected = path.join(root, "expected.json");
+  const observed = path.join(root, "observed.json");
+  fs.writeFileSync(expected, '["active"]');
+  const script = path.join(installed, "compare-discovery.mjs");
+  for (const [input, status] of [['["active"]', 0], ['["template"]', 1]]) {
+    fs.writeFileSync(observed, input);
+    const result = spawnSync(process.execPath, [script, expected, observed], { encoding: "utf8", windowsHide: true });
+    assert.equal(result.status, status, result.stderr);
+    assert.equal(JSON.parse(result.stdout).match, status === 0);
+  }
+  assert.equal(spawnSync(process.execPath, [script, expected, path.join(root, "absent.json")], { windowsHide: true }).status, 2);
+});