Add a readiness auditor (tools/pack-audit + Claude Code agent) that scores packs against the release-readiness checklist #49

Problem

docs/release-readiness-checklist.md is the promotion rubric for every pack, but applying it is manual: someone reads the repo, ticks boxes, and argues about edge cases. Packs built outside the org (vendor-delivered charts that must land on a Nebari cluster) make this worse, because they rarely start from this template and the gaps are structural rather than cosmetic.

We needed a repeatable answer to "how far is this pack from Alpha / Beta / GA, and what exactly do we change next?", and nothing in the org does that today (no tools/, no pack-audit workflow, no agent or skill; checked 2026-10-09).

Proposal

Add tools/pack-audit/ to this repo:

• checklist.yaml: the release-readiness checklist as data (id, level tag, category, check type auto|judgment|manual, applicability, expectation, and a source/example link per item pointing at the rule and at a first-party pack file that satisfies it).
• audit.py: a deterministic scanner and scorer. Renders the chart with the NebariApp toggle on and off, inspects the manifests (probes, securityContext, image pinning, NetworkPolicy, scrape annotations, literal secrets), validates the NebariApp spec against the operator CRD, validates pack-metadata.yaml against the dashboard schema, runs kubeconform (or the pure-Python kubernetes-validate) over the render, greps docs for the checklist's requirements, and writes reports/<pack>.{json,md} with a weighted score, the achieved level, and blockers per level. summary and summary_html.py compare several packs.
• .claude/agents/software-pack-auditor.md: a Claude Code agent that runs the scanner, reads the pack, decides the items that need a reader (README intent, auth docs, values reference, telemetry justification, appVersion correctness), rescores, and writes a verdict with a file-level fix list. Audit mode is read-only on the pack; fix mode is opt-in.
• reference/template-conventions.md: what a healthy pack looks like, distilled from this template and the first-party packs.
• docs/superpowers/specs/2026-10-09-pack-audit-design.md: how the rubric, scoring, and agent were derived.

How it was designed

1. This repo is the source of truth: the checklist, README, CRD reference, docs-site requirements, the five examples, and the CI workflows.
2. Eight first-party packs (llm-serving, mlflow, superset, harbor, langfuse, data-science, lgtm, nebari-nexus) were read for the conventions mature packs actually follow; equivalent deviations (hand-written NebariApp templates, GitLab CI) are accepted when they deliver the same guarantee.
3. nebari-operator nebariapp_types.go and nebariapp_controller.go at v0.1.1 supplied the real field set and runtime semantics.
4. Every checklist row is auto (scanner), judgment (agent with evidence), or manual (cluster/person, never scored), so scores are reproducible and a human can run the scanner alone.
5. Scored by promotion gate, weighted E=4/A=3/B=2/GA=1; first-party-only rows are N/A for packs outside the org; pre-sales and sign-off rows are listed but never scored.
6. Calibrated on examples/* and on data-science-pack (declared beta), which scores 78.7% and lands at Experimental by the letter of the rubric; the gap is mostly where content lives (README headings, examples/), plus a stale appVersion and no metrics story.

Things found along the way that this repo should fix independently

• examples/basic-nginx and examples/wrap-existing-chart omit routing in values.yaml. On operator v0.1.1 a NebariApp with spec.routing omitted gets no HTTPRoute (nebariapp_controller.go cleans it up) and TLS is treated as disabled. The README's annotated example and every first-party pack set routing explicitly.
• docs/nebariapp-crd-reference.md tracks alpha.19 and lacks routing.tls.secretName and landingPage.iconLight/iconDark, which the operator accepts now.
• The Telemetry row "ServiceMonitor or PodMonitor exposed" names a mechanism the NIC OpenTelemetry collector does not read: it discovers pods through prometheus.io/scrape annotations (see lgtm-pack values.yaml) or OTLP push. Suggest rewording to "metrics discoverable by the platform collector (scrape annotations or OTLP)".
• "Example values file deploys without modification (other than hostname)" has no machine-checkable definition; the auditor treats <...>/REPLACE_/CHANGEME placeholders as PARTIAL.

Open questions

• Should lint.yaml run audit.py scan examples/auth-fastapi as a self-test?
• Should pack-metadata.yaml grow last_audit_score / last_audit_at for the dashboard?
• Is tools/ the right home, or should the agent live somewhere every pack repo can invoke it?